[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:54 dist-test-slave-8vcz krb5kdc[16635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:54 dist-test-slave-8vcz krb5kdc[16635](info): set up 2 sockets
Jun 19 11:04:54 dist-test-slave-8vcz krb5kdc[16635](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 11:04:55.745640 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:04:55 dist-test-slave-8vcz krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:04:55.864970 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:39075
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:39075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:55.974493 16648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:55.974728 16648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:55.974766 16648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:55.978562 16648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:55.978606 16648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:55.978626 16648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:55.978641 16648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:55.978655 16648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:55.983626 16648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:39075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:39075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:55.984544 16648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:55.985415 16648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:55.993701 16654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:55.993706 16648 server_base.cc:1034] running on GCE node
W20260619 11:04:55.993727 16653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:55.993737 16656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:55.994489 16648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:55.995474 16648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:55.996661 16648 hybrid_clock.cc:648] HybridClock initialized: now 1781867095996656 us; error 33 us; skew 500 ppm
Jun 19 11:04:55 dist-test-slave-8vcz krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867095, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:55.999320 16648 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:04:56.000167 16648 webserver.cc:460] Webserver started at http://127.16.61.62:42733/ using document root <none> and password file <none>
I20260619 11:04:56.000651 16648 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:56.000705 16648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:56.000857 16648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:56.002715 16648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "bdf8f649590d483da2859e8393a7348e"
format_stamp: "Formatted at 2026-06-19 11:04:55 on dist-test-slave-8vcz"
server_key: "35be9e2250d8c10de8fc8a24ca94413a"
server_key_iv: "1b062e3d0ac0e322822758b0232900cf"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.003203 16648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "bdf8f649590d483da2859e8393a7348e"
format_stamp: "Formatted at 2026-06-19 11:04:55 on dist-test-slave-8vcz"
server_key: "35be9e2250d8c10de8fc8a24ca94413a"
server_key_iv: "1b062e3d0ac0e322822758b0232900cf"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.006629 16648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:04:56.008876 16663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.010023 16648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:04:56.010143 16648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "bdf8f649590d483da2859e8393a7348e"
format_stamp: "Formatted at 2026-06-19 11:04:55 on dist-test-slave-8vcz"
server_key: "35be9e2250d8c10de8fc8a24ca94413a"
server_key_iv: "1b062e3d0ac0e322822758b0232900cf"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.010238 16648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:56.020113 16648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:56.023123 16648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:56.023337 16648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:56.031526 16648 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:39075
I20260619 11:04:56.031559 16715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:39075 every 8 connection(s)
I20260619 11:04:56.032572 16648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:56.035331 16716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:56.040759 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 16648
I20260619 11:04:56.040941 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:04:56.041004 16716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e: Bootstrap starting.
I20260619 11:04:56.042282 16628 external_mini_cluster.cc:1428] Setting key 1f94b4087af2eb27c2d6a00ee0be6b10
I20260619 11:04:56.043464 16716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:56.044303 16716 log.cc:824] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:56.046563 16716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e: No bootstrap required, opened a new log
I20260619 11:04:56.050530 16716 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf8f649590d483da2859e8393a7348e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39075 } }
I20260619 11:04:56.050772 16716 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:56.050830 16716 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdf8f649590d483da2859e8393a7348e, State: Initialized, Role: FOLLOWER
I20260619 11:04:56.051296 16716 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf8f649590d483da2859e8393a7348e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39075 } }
I20260619 11:04:56.051455 16716 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:56.051548 16716 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:56.051694 16716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:56.052551 16716 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf8f649590d483da2859e8393a7348e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39075 } }
I20260619 11:04:56.052891 16716 leader_election.cc:302] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdf8f649590d483da2859e8393a7348e; no voters:
I20260619 11:04:56.053246 16716 leader_election.cc:288] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:56.053346 16721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:56.053558 16721 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [term 1 LEADER]: Becoming Leader. State: Replica: bdf8f649590d483da2859e8393a7348e, State: Running, Role: LEADER
I20260619 11:04:56.053843 16721 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf8f649590d483da2859e8393a7348e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39075 } }
I20260619 11:04:56.054471 16716 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:56.055514 16723 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdf8f649590d483da2859e8393a7348e. Latest consensus state: current_term: 1 leader_uuid: "bdf8f649590d483da2859e8393a7348e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf8f649590d483da2859e8393a7348e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39075 } } }
I20260619 11:04:56.055646 16723 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:56.056061 16722 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdf8f649590d483da2859e8393a7348e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdf8f649590d483da2859e8393a7348e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39075 } } }
I20260619 11:04:56.056140 16722 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e [sys.catalog]: This master's current role is: LEADER
W20260619 11:04:56.059633 16737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:04:56.059746 16737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:04:56.059821 16730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:56.060619 16730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:04:56.067974 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.045342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.16.61.62:39075)
0619 11:04:56.045704 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:56.045719 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:56.046983 (+ 1264us) server_negotiation.cc:407] Connection header received
0619 11:04:56.050161 (+ 3178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:56.050176 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:56.050487 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:56.050800 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:56.053134 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.054709 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.056572 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.057269 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.057426 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.061544 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:56.061556 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:56.061568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:56.061594 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:56.064222 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.064882 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.064889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.064895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.065024 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.065402 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.065406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.065408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.065813 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:56.066034 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:56.066679 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 11:04:56.066956 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":109,"threads_started":1}
I20260619 11:04:56.069258 16730 catalog_manager.cc:1347] Generated new cluster ID: 384c1e5391904e6299da4cd8c206f532
I20260619 11:04:56.069316 16730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:56.086117 16730 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:56.086969 16730 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:56.091595 16730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdf8f649590d483da2859e8393a7348e: Generated new TSK 0
I20260619 11:04:56.092272 16730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:04:56.130502 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39075
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:56.244990 16742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:56.245246 16742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:56.245289 16742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:56.249209 16742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:56.249264 16742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:56.249318 16742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:04:56.254195 16742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39075
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:56.255162 16742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:56.256235 16742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:56.265317 16747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:56.265326 16750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:56.265333 16748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:56.265491 16742 server_base.cc:1034] running on GCE node
I20260619 11:04:56.265976 16742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:56.266548 16742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:56.267757 16742 hybrid_clock.cc:648] HybridClock initialized: now 1781867096267743 us; error 36 us; skew 500 ppm
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:56.270483 16742 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:04:56.271272 16742 webserver.cc:460] Webserver started at http://127.16.61.1:39179/ using document root <none> and password file <none>
I20260619 11:04:56.271822 16742 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:56.271885 16742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:56.272039 16742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:56.273901 16742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "6a45dba5ca6342eab815df8f24defeff"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "6c9648a5d15759fe6da2e49c000c0d67"
server_key_iv: "8802fe8f097ba04540d439b28b91972e"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.274384 16742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "6a45dba5ca6342eab815df8f24defeff"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "6c9648a5d15759fe6da2e49c000c0d67"
server_key_iv: "8802fe8f097ba04540d439b28b91972e"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.278489 16742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:04:56.281293 16757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.282456 16742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:04:56.282544 16742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "6a45dba5ca6342eab815df8f24defeff"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "6c9648a5d15759fe6da2e49c000c0d67"
server_key_iv: "8802fe8f097ba04540d439b28b91972e"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.282629 16742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:56.294147 16742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:56.297238 16742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:56.297377 16742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:56.297868 16742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:56.298574 16742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:56.298619 16742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.298655 16742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:56.298679 16742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.307755 16742 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:35751
I20260619 11:04:56.307822 16870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:35751 every 8 connection(s)
I20260619 11:04:56.308593 16742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:04:56.316568 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 16742
I20260619 11:04:56.316689 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:04:56.316982 16628 external_mini_cluster.cc:1428] Setting key 46bc628ffb7d73d44788ceb62a26274d
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:04:56.322324 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.310390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:60907 (local address 127.16.61.62:39075)
0619 11:04:56.310525 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:56.310529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:56.311602 (+ 1073us) server_negotiation.cc:407] Connection header received
0619 11:04:56.312441 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:56.312444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:56.312490 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:56.312570 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:56.313698 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.314333 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.315455 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.315774 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.315865 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.318492 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:56.318495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:56.318497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:56.318516 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:56.319971 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.320528 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.320532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.320536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.320628 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.320989 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.320992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.320994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.321234 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:56.321346 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:56.321946 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 11:04:56.322081 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:04:56.323086 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.310779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:39075 (local address 127.16.61.1:60907)
0619 11:04:56.311455 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:56.311493 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:56.312256 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:56.312717 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:56.312723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:56.313073 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:56.313574 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:56.313586 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.314491 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.314494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:56.315336 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:56.315344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.315921 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.315928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:56.316128 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.316821 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:56.316836 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:56.318321 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:56.320119 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:56.320128 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:56.320146 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:56.320435 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:56.320725 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:56.320727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:56.320728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:56.320871 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:56.321345 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:56.321351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:56.321656 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 11:04:56.321874 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 11:04:56.322180 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":116,"threads_started":1}
I20260619 11:04:56.324437 16871 heartbeater.cc:347] Connected to a master server at 127.16.61.62:39075
I20260619 11:04:56.324679 16871 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:56.325228 16871 heartbeater.cc:511] Master 127.16.61.62:39075 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:04:56.326809 16680 ts_manager.cc:194] Registered new tserver with Master: 6a45dba5ca6342eab815df8f24defeff (127.16.61.1:35751)
I20260619 11:04:56.328034 16680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:60907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:04:56.343014 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39075
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:56.452414 16876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:56.452634 16876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:56.452674 16876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:56.456768 16876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:56.456818 16876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:56.456903 16876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:04:56.461826 16876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39075
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:56.462723 16876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:56.463641 16876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:56.472508 16882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:56.472497 16884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:56.472497 16881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:56.472772 16876 server_base.cc:1034] running on GCE node
I20260619 11:04:56.473232 16876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:56.473809 16876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:56.474988 16876 hybrid_clock.cc:648] HybridClock initialized: now 1781867096474975 us; error 34 us; skew 500 ppm
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:56.477741 16876 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:04:56.478515 16876 webserver.cc:460] Webserver started at http://127.16.61.2:44057/ using document root <none> and password file <none>
I20260619 11:04:56.479030 16876 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:56.479087 16876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:56.479287 16876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:56.481268 16876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "336f161be8ea45eca6be3c391cd9a1af"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "c583f8261cc4f6f1adf026c826989813"
server_key_iv: "dd196de6de0de9981b443e3b6f3081b7"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.481797 16876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "336f161be8ea45eca6be3c391cd9a1af"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "c583f8261cc4f6f1adf026c826989813"
server_key_iv: "dd196de6de0de9981b443e3b6f3081b7"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.485244 16876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 11:04:56.487550 16891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.488735 16876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 11:04:56.488862 16876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "336f161be8ea45eca6be3c391cd9a1af"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "c583f8261cc4f6f1adf026c826989813"
server_key_iv: "dd196de6de0de9981b443e3b6f3081b7"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.488962 16876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:56.519797 16876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:56.522943 16876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:56.523130 16876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:56.523749 16876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:56.524541 16876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:56.524587 16876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.524653 16876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:56.524688 16876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.533311 16876 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:35737
I20260619 11:04:56.533341 17004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:35737 every 8 connection(s)
I20260619 11:04:56.534107 16876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:04:56.538517 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 16876
I20260619 11:04:56.538648 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:04:56.538906 16628 external_mini_cluster.cc:1428] Setting key efa9d20c36eedcdb87da0ce20cb2b239
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:04:56.548158 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.535738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:47145 (local address 127.16.61.62:39075)
0619 11:04:56.535870 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:56.535873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:56.536854 (+ 981us) server_negotiation.cc:407] Connection header received
0619 11:04:56.537713 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:56.537717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:56.537776 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:56.537859 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:56.539058 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.539736 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.540928 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.541154 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.541258 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.544238 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:56.544244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:56.544251 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:56.544283 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:56.546066 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.546580 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.546585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.546587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.546679 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.546994 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.546999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.547004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.547255 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:56.547366 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:56.547845 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 11:04:56.547963 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
I20260619 11:04:56.548875 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.536093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:39075 (local address 127.16.61.2:47145)
0619 11:04:56.536719 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:56.536756 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:56.537512 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:56.538025 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:56.538031 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:56.538387 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:56.538904 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:56.538918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.539855 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.539859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:56.540803 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:56.540813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.541292 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.541295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:56.541482 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.542433 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:56.542457 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:56.544069 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:56.546186 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:56.546191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:56.546229 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:56.546477 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:56.546782 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:56.546784 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:56.546785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:56.546903 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:56.547364 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:56.547369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:56.547584 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 11:04:56.547786 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 11:04:56.548022 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":120,"threads_started":1}
I20260619 11:04:56.549990 17005 heartbeater.cc:347] Connected to a master server at 127.16.61.62:39075
I20260619 11:04:56.550189 17005 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:56.550660 17005 heartbeater.cc:511] Master 127.16.61.62:39075 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:04:56.551712 16680 ts_manager.cc:194] Registered new tserver with Master: 336f161be8ea45eca6be3c391cd9a1af (127.16.61.2:35737)
I20260619 11:04:56.552290 16680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:47145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:04:56.566218 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39075
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:56.673333 17010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:56.673554 17010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:56.673595 17010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:56.677428 17010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:56.677474 17010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:56.677529 17010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:04:56.682606 17010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39075
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:56.683459 17010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:56.684386 17010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:56.693208 17016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:56.693240 17018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:56.693240 17015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:56.693346 17010 server_base.cc:1034] running on GCE node
I20260619 11:04:56.693853 17010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:56.694463 17010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:56.695626 17010 hybrid_clock.cc:648] HybridClock initialized: now 1781867096695596 us; error 40 us; skew 500 ppm
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:56.698105 17010 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:04:56.698865 17010 webserver.cc:460] Webserver started at http://127.16.61.3:35197/ using document root <none> and password file <none>
I20260619 11:04:56.699426 17010 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:56.699517 17010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:56.699793 17010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:56.701736 17010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "6a67f40e4cba4172ae0d6689064f3fcc"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "c91fa39483a977bd0627c1eb6bb9f96e"
server_key_iv: "33e864e1696111abc974a801fef067a4"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.702283 17010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "6a67f40e4cba4172ae0d6689064f3fcc"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "c91fa39483a977bd0627c1eb6bb9f96e"
server_key_iv: "33e864e1696111abc974a801fef067a4"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.705744 17010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:04:56.708000 17025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.709052 17010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:04:56.709189 17010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "6a67f40e4cba4172ae0d6689064f3fcc"
format_stamp: "Formatted at 2026-06-19 11:04:56 on dist-test-slave-8vcz"
server_key: "c91fa39483a977bd0627c1eb6bb9f96e"
server_key_iv: "33e864e1696111abc974a801fef067a4"
server_key_version: "encryptionkey@0"
I20260619 11:04:56.709302 17010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:56.724916 17010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:56.727567 17010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:56.727805 17010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:56.728336 17010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:56.729077 17010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:56.729147 17010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.729218 17010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:56.729267 17010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:56.738138 17010 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:33567
I20260619 11:04:56.738171 17138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:33567 every 8 connection(s)
I20260619 11:04:56.738989 17010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:04:56.741981 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 17010
I20260619 11:04:56.742086 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:04:56.742357 16628 external_mini_cluster.cc:1428] Setting key e33589bea9835d972c0debc14193d344
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:04:56.751571 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.740669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:56553 (local address 127.16.61.62:39075)
0619 11:04:56.740794 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:56.740798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:56.741748 (+ 950us) server_negotiation.cc:407] Connection header received
0619 11:04:56.742990 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:56.742993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:56.743038 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:56.743140 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:56.744532 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.745138 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.746192 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.746398 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.746470 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.748690 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:56.748693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:56.748695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:56.748716 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:56.749938 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.750379 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.750381 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.750382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.750453 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.750718 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.750720 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.750721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.750869 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:56.750940 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:56.751372 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 11:04:56.751461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:04:56.752329 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.741000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:39075 (local address 127.16.61.3:56553)
0619 11:04:56.741589 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:56.741627 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:56.742612 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:56.743464 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:56.743470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:56.743924 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:56.744377 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:56.744390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.745260 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.745263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:56.746093 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:56.746102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.746514 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.746516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:56.746652 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.747140 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:56.747155 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:56.748542 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:56.750043 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:56.750048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:56.750057 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:56.750295 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:56.750527 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:56.750529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:56.750530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:56.750646 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:56.750949 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:56.750954 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:56.751146 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 11:04:56.751310 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 11:04:56.751526 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":104,"threads_started":1}
I20260619 11:04:56.753417 17139 heartbeater.cc:347] Connected to a master server at 127.16.61.62:39075
I20260619 11:04:56.753607 17139 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:56.754053 17139 heartbeater.cc:511] Master 127.16.61.62:39075 requested a full tablet report, sending...
I20260619 11:04:56.754952 16680 ts_manager.cc:194] Registered new tserver with Master: 6a67f40e4cba4172ae0d6689064f3fcc (127.16.61.3:33567)
I20260619 11:04:56.755745 16680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:56553
I20260619 11:04:56.756564 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:04:56.780756 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.772171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.16.61.1:35751)
0619 11:04:56.772488 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:56.772498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:56.772519 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:04:56.772564 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:56.772569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:56.772726 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:56.772840 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:56.773463 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.774189 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.774967 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.775326 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.775408 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.777642 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:56.777650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:56.777662 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:56.777683 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:56.779163 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.779540 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.779545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.779546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.779620 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.779894 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.779896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.779898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.780104 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:56.780265 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:56.780405 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 11:04:56.780549 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":134,"threads_started":1}
W20260619 11:04:56.781719 16785 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51382
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[16635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:04:56.806056 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:56.798041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.16.61.1:35751)
0619 11:04:56.798190 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:56.798195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:56.798374 (+ 179us) server_negotiation.cc:407] Connection header received
0619 11:04:56.798542 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:56.798545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:56.798584 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:56.798667 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:56.799302 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.799857 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.800672 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:56.800915 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:56.801024 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:56.803274 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:56.803277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:56.803279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:56.803300 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:56.804648 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.805038 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.805040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.805041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.805124 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:56.805367 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:56.805370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:56.805371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:56.805523 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:56.805624 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:56.805854 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 11:04:56.805946 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:04:56.807042 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 16742
I20260619 11:04:56.812501 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 16876
I20260619 11:04:56.817633 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 17010
I20260619 11:04:56.822700 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 16648
2026-06-19T11:04:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2140 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[17153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[17153](info): set up 2 sockets
Jun 19 11:04:56 dist-test-slave-8vcz krb5kdc[17153](info): commencing operation
krb5kdc: starting...
W20260619 11:04:57.855793 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:04:57 dist-test-slave-8vcz krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:04:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:04:57.968573 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:42177
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:58.082382 17166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:58.082615 17166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:58.082653 17166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:58.088900 17166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:04:58.088970 17166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:58.089001 17166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:58.089035 17166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:04:58.089064 17166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:04:58.097847 17166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:42177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:58.099186 17166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:58.100499 17166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:58.110483 17174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:58.110486 17171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:58.110478 17172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:58.110700 17166 server_base.cc:1034] running on GCE node
I20260619 11:04:58.111542 17166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:58.112995 17166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:58.114178 17166 hybrid_clock.cc:648] HybridClock initialized: now 1781867098114151 us; error 47 us; skew 500 ppm
Jun 19 11:04:58 dist-test-slave-8vcz krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:58.116868 17166 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:04:58.117654 17166 webserver.cc:460] Webserver started at http://127.16.61.62:42387/ using document root <none> and password file <none>
I20260619 11:04:58.118177 17166 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:58.118237 17166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:58.118428 17166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:58.120523 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "a7b959a721e9486bb797e633a21d8285"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "5f4b912c2fe3a2a8a9b7059f3288c556"
server_key_iv: "3f1ab4986517261fc6ec12396729b680"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.121057 17166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "a7b959a721e9486bb797e633a21d8285"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "5f4b912c2fe3a2a8a9b7059f3288c556"
server_key_iv: "3f1ab4986517261fc6ec12396729b680"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.124540 17166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:58.126741 17181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.128043 17166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:58.128175 17166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "a7b959a721e9486bb797e633a21d8285"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "5f4b912c2fe3a2a8a9b7059f3288c556"
server_key_iv: "3f1ab4986517261fc6ec12396729b680"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.128264 17166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:58.148335 17166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:58.150921 17166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:58.151242 17166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:58.157917 17166 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:42177
I20260619 11:04:58.157943 17233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:42177 every 8 connection(s)
I20260619 11:04:58.158789 17166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:04:58.161312 17234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:58.163926 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 17166
I20260619 11:04:58.164039 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:04:58.164305 16628 external_mini_cluster.cc:1428] Setting key 7561bb0605c98882839d2fb518a2ef7c
I20260619 11:04:58.166919 17234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285: Bootstrap starting.
I20260619 11:04:58.168959 17234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:58.169632 17234 log.cc:824] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:58.171758 17234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285: No bootstrap required, opened a new log
Jun 19 11:04:58 dist-test-slave-8vcz krb5kdc[17153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:04:58.176133 17234 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b959a721e9486bb797e633a21d8285" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42177 } }
I20260619 11:04:58.176400 17234 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:58.176465 17234 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b959a721e9486bb797e633a21d8285, State: Initialized, Role: FOLLOWER
I20260619 11:04:58.176905 17234 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b959a721e9486bb797e633a21d8285" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42177 } }
I20260619 11:04:58.177063 17234 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:04:58.177137 17234 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:04:58.177234 17234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:58.178198 17234 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b959a721e9486bb797e633a21d8285" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42177 } }
I20260619 11:04:58.178525 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.165651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.16.61.62:42177)
0619 11:04:58.166058 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:58.166072 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:58.166105 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:04:58.166726 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:58.166742 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:58.167099 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:58.167453 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:58.168257 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.169250 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.170204 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.170828 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.170996 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.173590 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:58.173597 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:58.173613 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:58.173640 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:58.175551 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.176014 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.176019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.176024 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.176158 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.176421 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.176425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.176427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.176872 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:58.177070 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:58.177342 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 11:04:58.177612 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260619 11:04:58.178525 17234 leader_election.cc:302] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7b959a721e9486bb797e633a21d8285; no voters:
I20260619 11:04:58.178858 17234 leader_election.cc:288] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:04:58.178960 17239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:58.179199 17239 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [term 1 LEADER]: Becoming Leader. State: Replica: a7b959a721e9486bb797e633a21d8285, State: Running, Role: LEADER
I20260619 11:04:58.179488 17239 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b959a721e9486bb797e633a21d8285" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42177 } }
I20260619 11:04:58.179993 17234 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:04:58.181012 17241 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7b959a721e9486bb797e633a21d8285. Latest consensus state: current_term: 1 leader_uuid: "a7b959a721e9486bb797e633a21d8285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b959a721e9486bb797e633a21d8285" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42177 } } }
I20260619 11:04:58.181142 17241 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:58.181478 17240 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7b959a721e9486bb797e633a21d8285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b959a721e9486bb797e633a21d8285" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42177 } } }
I20260619 11:04:58.181553 17248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:04:58.181567 17240 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285 [sys.catalog]: This master's current role is: LEADER
I20260619 11:04:58.184819 17248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:04:58.189972 17248 catalog_manager.cc:1347] Generated new cluster ID: 46941d2de3a748f3a131fc318c668976
I20260619 11:04:58.190032 17248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:04:58.208215 17248 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:04:58.209342 17248 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:04:58.220938 17248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7b959a721e9486bb797e633a21d8285: Generated new TSK 0
I20260619 11:04:58.221832 17248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:04:58.254097 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42177
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:58.360395 17260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:58.360623 17260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:58.360664 17260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:58.364447 17260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:58.364492 17260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:58.364548 17260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:04:58.369561 17260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:58.370429 17260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:58.371299 17260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:58.383575 17266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:58.383548 17265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:58.383538 17268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:58.383985 17260 server_base.cc:1034] running on GCE node
I20260619 11:04:58.384430 17260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:58.384972 17260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:58.386163 17260 hybrid_clock.cc:648] HybridClock initialized: now 1781867098386132 us; error 40 us; skew 500 ppm
Jun 19 11:04:58 dist-test-slave-8vcz krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:58.388932 17260 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:04:58.389753 17260 webserver.cc:460] Webserver started at http://127.16.61.1:46093/ using document root <none> and password file <none>
I20260619 11:04:58.390292 17260 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:58.390378 17260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:58.390594 17260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:58.392577 17260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "e4784fe2a1dd449482822ddd750a5551"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "356ea30ac74e7a074f7d3e717a4a6484"
server_key_iv: "a317c6f23cabc2e70d9a3b324fc74638"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.393119 17260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "e4784fe2a1dd449482822ddd750a5551"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "356ea30ac74e7a074f7d3e717a4a6484"
server_key_iv: "a317c6f23cabc2e70d9a3b324fc74638"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.396512 17260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:04:58.398710 17275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.399907 17260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:04:58.400029 17260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "e4784fe2a1dd449482822ddd750a5551"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "356ea30ac74e7a074f7d3e717a4a6484"
server_key_iv: "a317c6f23cabc2e70d9a3b324fc74638"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.400137 17260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:58.423280 17260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:58.425925 17260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:58.426113 17260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:58.426652 17260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:58.427435 17260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:58.427506 17260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.427572 17260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:58.427618 17260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.436527 17260 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:33137
I20260619 11:04:58.436558 17388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:33137 every 8 connection(s)
I20260619 11:04:58.437355 17260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:04:58.439497 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 17260
I20260619 11:04:58.439612 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:04:58.439924 16628 external_mini_cluster.cc:1428] Setting key 1f448920ed64502d6557145b50604eae
Jun 19 11:04:58 dist-test-slave-8vcz krb5kdc[17153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:04:58.450522 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.439001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:60739 (local address 127.16.61.62:42177)
0619 11:04:58.439132 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:58.439135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:58.439917 (+ 782us) server_negotiation.cc:407] Connection header received
0619 11:04:58.441181 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:58.441184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:58.441236 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:58.441318 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:58.442414 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.442942 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.444114 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.444349 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.444461 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.446747 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:58.446755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:58.446757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:58.446778 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:58.448535 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.449048 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.449056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.449057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.449126 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.449455 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.449460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.449462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.449686 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:58.449790 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:58.450241 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 11:04:58.450357 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
I20260619 11:04:58.451414 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.439308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42177 (local address 127.16.61.1:60739)
0619 11:04:58.439761 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:58.439810 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:58.440972 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:58.441499 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:58.441506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:58.441836 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:58.442266 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.442278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.443062 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.443067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.443994 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.444003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.444492 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.444496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.444659 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.445217 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:58.445233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:58.446581 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:58.448649 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:58.448655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:58.448668 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:58.448933 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:58.449217 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:58.449221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:58.449223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:58.449359 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:58.449786 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:58.449792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:58.450022 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 11:04:58.450203 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 11:04:58.450534 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":134,"threads_started":1}
I20260619 11:04:58.452718 17389 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42177
I20260619 11:04:58.452950 17389 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:58.453402 17389 heartbeater.cc:511] Master 127.16.61.62:42177 requested a full tablet report, sending...
I20260619 11:04:58.454954 17198 ts_manager.cc:194] Registered new tserver with Master: e4784fe2a1dd449482822ddd750a5551 (127.16.61.1:33137)
I20260619 11:04:58.456067 17198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:60739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:04:58.466871 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42177
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:58.573222 17394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:58.573475 17394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:58.573549 17394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:58.577476 17394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:58.577545 17394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:58.577626 17394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:04:58.582917 17394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:58.583900 17394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:58.584930 17394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:58.593855 17400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:58.593760 17399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:58.593768 17402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:58.594177 17394 server_base.cc:1034] running on GCE node
I20260619 11:04:58.594508 17394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:58.595055 17394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:58.596231 17394 hybrid_clock.cc:648] HybridClock initialized: now 1781867098596210 us; error 30 us; skew 500 ppm
Jun 19 11:04:58 dist-test-slave-8vcz krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:58.598909 17394 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:04:58.599815 17394 webserver.cc:460] Webserver started at http://127.16.61.2:45813/ using document root <none> and password file <none>
I20260619 11:04:58.600342 17394 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:58.600433 17394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:58.600649 17394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:58.602528 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "28f18b480560439e9d8ec8783e37518c"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "1795144d339ed57a223b4f54dcd8ace4"
server_key_iv: "2cc5f20ed66e48fce38cafc0ca870c5f"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.603073 17394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "28f18b480560439e9d8ec8783e37518c"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "1795144d339ed57a223b4f54dcd8ace4"
server_key_iv: "2cc5f20ed66e48fce38cafc0ca870c5f"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.606513 17394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:04:58.608861 17409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.610003 17394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:04:58.610145 17394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "28f18b480560439e9d8ec8783e37518c"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "1795144d339ed57a223b4f54dcd8ace4"
server_key_iv: "2cc5f20ed66e48fce38cafc0ca870c5f"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.610265 17394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:58.639222 17394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:58.642549 17394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:58.642747 17394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:58.643303 17394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:58.644147 17394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:58.644224 17394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.644312 17394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:58.644371 17394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.652809 17394 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:43241
I20260619 11:04:58.652837 17522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:43241 every 8 connection(s)
I20260619 11:04:58.653661 17394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:04:58.662326 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 17394
I20260619 11:04:58.662416 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
Jun 19 11:04:58 dist-test-slave-8vcz krb5kdc[17153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:04:58.662704 16628 external_mini_cluster.cc:1428] Setting key 3dbf3e6719b4ff500811657ef6f286ce
I20260619 11:04:58.666952 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.655383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:44609 (local address 127.16.61.62:42177)
0619 11:04:58.655510 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:58.655514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:58.656295 (+ 781us) server_negotiation.cc:407] Connection header received
0619 11:04:58.657567 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:58.657570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:58.657627 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:58.657707 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:58.658794 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.659325 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.660498 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.660734 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.660847 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.663207 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:58.663213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:58.663216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:58.663241 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:58.664738 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.665341 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.665344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.665345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.665418 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.665809 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.665815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.665818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.666053 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:58.666164 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:58.666607 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 11:04:58.666732 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:04:58.667567 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.655745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42177 (local address 127.16.61.2:44609)
0619 11:04:58.656145 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:58.656185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:58.657324 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:58.657864 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:58.657871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:58.658185 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:58.658646 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.658658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.659449 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.659454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.660369 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.660378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.660879 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.660884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.661043 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.661587 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:58.661603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:58.663049 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:58.664867 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:58.664876 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:58.664893 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:58.665244 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:58.665506 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:58.665512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:58.665517 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:58.665699 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:58.666160 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:58.666164 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:58.666373 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 11:04:58.666521 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 11:04:58.666740 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
I20260619 11:04:58.668671 17523 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42177
I20260619 11:04:58.668907 17523 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:58.669340 17523 heartbeater.cc:511] Master 127.16.61.62:42177 requested a full tablet report, sending...
I20260619 11:04:58.670269 17198 ts_manager.cc:194] Registered new tserver with Master: 28f18b480560439e9d8ec8783e37518c (127.16.61.2:43241)
I20260619 11:04:58.670881 17198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:44609
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:04:58.690114 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42177
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:04:58.794926 17528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:04:58.795153 17528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:04:58.795194 17528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:04:58.799145 17528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:04:58.799197 17528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:04:58.799296 17528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:04:58.804409 17528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:04:58.805356 17528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:04:58.806281 17528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:04:58.815356 17533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:04:58.815385 17536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:58.815447 17528 server_base.cc:1034] running on GCE node
W20260619 11:04:58.815388 17534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:04:58.815961 17528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:04:58.816540 17528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:04:58.817745 17528 hybrid_clock.cc:648] HybridClock initialized: now 1781867098817734 us; error 29 us; skew 500 ppm
Jun 19 11:04:58 dist-test-slave-8vcz krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:04:58.820470 17528 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:04:58.821327 17528 webserver.cc:460] Webserver started at http://127.16.61.3:38839/ using document root <none> and password file <none>
I20260619 11:04:58.821874 17528 fs_manager.cc:359] Metadata directory not provided
I20260619 11:04:58.821962 17528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:04:58.822185 17528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:04:58.824183 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "2f3849e9676d4180a87eb28baf6f76f8"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "910e1a5d89bf109881b565111f3bc974"
server_key_iv: "9a6b6972ee9f8567edb4a11a19e2171d"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.824728 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "2f3849e9676d4180a87eb28baf6f76f8"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "910e1a5d89bf109881b565111f3bc974"
server_key_iv: "9a6b6972ee9f8567edb4a11a19e2171d"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.828342 17528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:04:58.830658 17543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.831859 17528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:04:58.831980 17528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "2f3849e9676d4180a87eb28baf6f76f8"
format_stamp: "Formatted at 2026-06-19 11:04:58 on dist-test-slave-8vcz"
server_key: "910e1a5d89bf109881b565111f3bc974"
server_key_iv: "9a6b6972ee9f8567edb4a11a19e2171d"
server_key_version: "encryptionkey@0"
I20260619 11:04:58.832082 17528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:04:58.849591 17528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:04:58.852262 17528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:04:58.852456 17528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:04:58.852974 17528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:04:58.853732 17528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:04:58.853802 17528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.853870 17528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:04:58.853911 17528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:04:58.862145 17528 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:40591
I20260619 11:04:58.862170 17656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:40591 every 8 connection(s)
I20260619 11:04:58.862939 17528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:04:58.865310 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 17528
I20260619 11:04:58.865420 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:04:58.865686 16628 external_mini_cluster.cc:1428] Setting key bb243077a3953ab2ab9f4f3b3511e35e
Jun 19 11:04:58 dist-test-slave-8vcz krb5kdc[17153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:04:58.875249 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.864614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:51671 (local address 127.16.61.62:42177)
0619 11:04:58.864747 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:58.864751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:58.865558 (+ 807us) server_negotiation.cc:407] Connection header received
0619 11:04:58.866812 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:58.866815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:58.866866 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:58.866943 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:58.868006 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.868527 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.869618 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.869840 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.869949 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.872119 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:58.872123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:58.872126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:58.872156 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:58.873390 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.873858 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.873863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.873865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.873940 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.874259 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.874264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.874265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.874428 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:58.874532 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:58.874970 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 11:04:58.875078 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:04:58.875988 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.864951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42177 (local address 127.16.61.3:51671)
0619 11:04:58.865397 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:58.865450 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:58.866610 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:58.867094 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:58.867100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:58.867428 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:04:58.867872 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.867884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.868648 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.868652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.869495 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.869503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.869977 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.869981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.870146 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.870604 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:04:58.870618 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:04:58.871966 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:04:58.873501 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:58.873506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:58.873519 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:58.873758 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:58.874030 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:04:58.874035 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:04:58.874037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:04:58.874162 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:04:58.874528 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:04:58.874534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:04:58.874757 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 11:04:58.874931 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 11:04:58.875206 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":151,"threads_started":1}
I20260619 11:04:58.876937 17657 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42177
I20260619 11:04:58.877154 17657 heartbeater.cc:464] Registering TS with master...
I20260619 11:04:58.877583 17657 heartbeater.cc:511] Master 127.16.61.62:42177 requested a full tablet report, sending...
I20260619 11:04:58.878593 17198 ts_manager.cc:194] Registered new tserver with Master: 2f3849e9676d4180a87eb28baf6f76f8 (127.16.61.3:40591)
I20260619 11:04:58.879141 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:04:58.879309 17198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:51671
I20260619 11:04:58.888854 17237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.882414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.16.61.62:42177)
0619 11:04:58.882513 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:58.882516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:58.882687 (+ 171us) server_negotiation.cc:407] Connection header received
0619 11:04:58.882777 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:58.882779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:58.882811 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:58.882866 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:58.883495 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.884012 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.884773 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.884964 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.885032 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.886201 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:58.886204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:58.886226 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:58.886245 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:58.887587 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.887963 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.887966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.887967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.888032 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:58.888259 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:58.888261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:58.888262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:58.888432 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:58.888526 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:58.888680 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 11:04:58.888758 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:04:58.892774 17198 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:04:58.895498 17198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:04:58.909453 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.903970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.16.61.3:40591)
0619 11:04:58.904550 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:58.904562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:58.904580 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:04:58.904650 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:58.904657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:58.904826 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:58.904964 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:58.905390 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.906546 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.908266 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.908801 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.908938 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.909075 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:58.909225 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 11:04:58.909309 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":75,"threads_started":1}
I20260619 11:04:58.909452 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.903509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40591 (local address 127.0.0.1:42486)
0619 11:04:58.904233 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:58.904248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:58.904352 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:58.904979 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:58.904985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:58.905008 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:58.905255 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.905264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.906681 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.906684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.908090 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.908096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.908946 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.908949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.909046 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.909080 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:04:58.909186 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:04:58.909301 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":281,"spinlock_wait_cycles":768,"thread_start_us":81,"threads_started":1}
I20260619 11:04:58.909508 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.903178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:43241 (local address 127.0.0.1:52158)
0619 11:04:58.904141 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:58.904152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:58.904232 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:58.905067 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:58.905069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:58.905081 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:58.905315 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.905345 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.906729 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.906732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.907924 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.907933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.909099 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.909102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.909195 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.909219 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:04:58.909268 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 11:04:58.909380 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":83,"spinlock_wait_cycles":11008,"thread_start_us":93,"threads_started":1}
I20260619 11:04:58.909647 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.903281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51582 (local address 127.16.61.1:33137)
0619 11:04:58.903997 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:58.904012 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:58.904331 (+ 319us) server_negotiation.cc:407] Connection header received
0619 11:04:58.904410 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:58.904415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:58.904611 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:58.904830 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:58.905444 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.906161 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.908034 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.908862 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.908970 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.909112 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:58.909401 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 11:04:58.909491 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":119,"threads_started":1}
I20260619 11:04:58.910252 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.903816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.16.61.2:43241)
0619 11:04:58.904688 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:58.904702 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:58.904724 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:04:58.904790 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:58.904793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:58.904957 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:58.905100 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:58.905475 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.906573 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.908216 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.908973 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.909862 (+ 889us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.909982 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:58.910032 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 11:04:58.910114 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":147,"threads_started":1}
I20260619 11:04:58.911882 17669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:58.903144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33137 (local address 127.0.0.1:51582)
0619 11:04:58.903900 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:58.904074 (+ 174us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:58.904198 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:58.904993 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:58.904999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:58.905015 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:58.905331 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.905341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.906295 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.906301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.907904 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:58.907916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:58.909159 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:58.909162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:58.909250 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:58.909276 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:04:58.909333 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 11:04:58.911797 (+ 2464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":2490,"thread_start_us":72,"threads_started":1}
I20260619 11:04:58.912940 17457 tablet_service.cc:1467] Processing CreateTablet for tablet 159789cc6662409b956d7d6736240b57 (DEFAULT_TABLE table=test-table [id=57ac347f1e444cc5849095fb7dfc5f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:58.912955 17591 tablet_service.cc:1467] Processing CreateTablet for tablet 159789cc6662409b956d7d6736240b57 (DEFAULT_TABLE table=test-table [id=57ac347f1e444cc5849095fb7dfc5f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:58.914004 17591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159789cc6662409b956d7d6736240b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:58.914004 17457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159789cc6662409b956d7d6736240b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:58.917773 17323 tablet_service.cc:1467] Processing CreateTablet for tablet 159789cc6662409b956d7d6736240b57 (DEFAULT_TABLE table=test-table [id=57ac347f1e444cc5849095fb7dfc5f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:04:58.918691 17323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159789cc6662409b956d7d6736240b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:04:58.921041 17674 tablet_bootstrap.cc:492] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8: Bootstrap starting.
I20260619 11:04:58.922109 17675 tablet_bootstrap.cc:492] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c: Bootstrap starting.
I20260619 11:04:58.923790 17674 tablet_bootstrap.cc:654] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:58.924141 17675 tablet_bootstrap.cc:654] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:58.924798 17675 log.cc:824] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:58.924971 17674 log.cc:824] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:58.926263 17678 tablet_bootstrap.cc:492] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551: Bootstrap starting.
I20260619 11:04:58.926726 17675 tablet_bootstrap.cc:492] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c: No bootstrap required, opened a new log
I20260619 11:04:58.926908 17675 ts_tablet_manager.cc:1397] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 11:04:58.928267 17678 tablet_bootstrap.cc:654] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551: Neither blocks nor log segments found. Creating new log.
I20260619 11:04:58.929176 17678 log.cc:824] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551: Log is configured to *not* fsync() on all Append() calls
I20260619 11:04:58.930371 17675 raft_consensus.cc:348] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:58.930637 17675 raft_consensus.cc:374] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:58.930694 17675 raft_consensus.cc:729] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f18b480560439e9d8ec8783e37518c, State: Initialized, Role: FOLLOWER
I20260619 11:04:58.930706 17678 tablet_bootstrap.cc:492] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551: No bootstrap required, opened a new log
I20260619 11:04:58.930784 17674 tablet_bootstrap.cc:492] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8: No bootstrap required, opened a new log
I20260619 11:04:58.930964 17674 ts_tablet_manager.cc:1397] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260619 11:04:58.930985 17678 ts_tablet_manager.cc:1397] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:04:58.931149 17675 consensus_queue.cc:260] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:58.931799 17523 heartbeater.cc:503] Master 127.16.61.62:42177 was elected leader, sending a full tablet report...
I20260619 11:04:58.932153 17675 ts_tablet_manager.cc:1428] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 11:04:58.934163 17674 raft_consensus.cc:348] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:58.934352 17674 raft_consensus.cc:374] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:58.934383 17674 raft_consensus.cc:729] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3849e9676d4180a87eb28baf6f76f8, State: Initialized, Role: FOLLOWER
I20260619 11:04:58.934803 17674 consensus_queue.cc:260] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:58.935137 17678 raft_consensus.cc:348] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:58.935447 17678 raft_consensus.cc:374] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:04:58.935506 17678 raft_consensus.cc:729] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4784fe2a1dd449482822ddd750a5551, State: Initialized, Role: FOLLOWER
I20260619 11:04:58.935772 17674 ts_tablet_manager.cc:1428] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 11:04:58.935964 17678 consensus_queue.cc:260] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:58.936084 17657 heartbeater.cc:503] Master 127.16.61.62:42177 was elected leader, sending a full tablet report...
I20260619 11:04:58.936786 17678 ts_tablet_manager.cc:1428] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:04:58.937134 17389 heartbeater.cc:503] Master 127.16.61.62:42177 was elected leader, sending a full tablet report...
W20260619 11:04:58.938344 17390 tablet.cc:2369] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:04:59.014081 17681 raft_consensus.cc:482] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:04:59.014279 17681 raft_consensus.cc:504] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:59.015410 17681 leader_election.cc:288] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4784fe2a1dd449482822ddd750a5551 (127.16.61.1:33137), 28f18b480560439e9d8ec8783e37518c (127.16.61.2:43241)
I20260619 11:04:59.019789 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:59.015612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33137 (local address 127.16.61.3:43831)
0619 11:04:59.015742 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:59.015758 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:59.015851 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:59.016158 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:59.016164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:59.016187 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:59.016319 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:59.016330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.017343 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.017346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:59.018545 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:59.018555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.019355 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.019358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:59.019463 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:59.019486 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:04:59.019563 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:04:59.019630 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 11:04:59.019824 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:59.015693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:43831 (local address 127.16.61.1:33137)
0619 11:04:59.015813 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:59.015817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:59.015831 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:04:59.015978 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:59.015981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:59.016033 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:59.016116 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:59.016442 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.017220 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.018661 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.019247 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.019351 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:59.019416 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:59.019596 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 11:04:59.019706 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:04:59.019872 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:59.015779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:35657 (local address 127.16.61.2:43241)
0619 11:04:59.015907 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:59.015910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:59.016423 (+ 513us) server_negotiation.cc:407] Connection header received
0619 11:04:59.016497 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:59.016500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:59.016545 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:59.016620 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:04:59.016826 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.017462 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.018791 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.019384 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.019478 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:59.019517 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:59.019708 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 11:04:59.019770 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:04:59.020125 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:59.015618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:43241 (local address 127.16.61.3:35657)
0619 11:04:59.016303 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 11:04:59.016316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:04:59.016417 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:04:59.016633 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:04:59.016635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:04:59.016647 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:04:59.016729 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:59.016733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.017549 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.017552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:59.018678 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:04:59.018687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.019481 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.019483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:04:59.019584 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:59.019602 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:04:59.019685 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 11:04:59.019741 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":341,"thread_start_us":125,"threads_started":1}
I20260619 11:04:59.020460 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159789cc6662409b956d7d6736240b57" candidate_uuid: "2f3849e9676d4180a87eb28baf6f76f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4784fe2a1dd449482822ddd750a5551" is_pre_election: true
I20260619 11:04:59.020794 17343 raft_consensus.cc:2454] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3849e9676d4180a87eb28baf6f76f8 in term 0.
I20260619 11:04:59.020752 17477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159789cc6662409b956d7d6736240b57" candidate_uuid: "2f3849e9676d4180a87eb28baf6f76f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f18b480560439e9d8ec8783e37518c" is_pre_election: true
I20260619 11:04:59.021025 17477 raft_consensus.cc:2454] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3849e9676d4180a87eb28baf6f76f8 in term 0.
I20260619 11:04:59.021284 17545 leader_election.cc:302] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3849e9676d4180a87eb28baf6f76f8, e4784fe2a1dd449482822ddd750a5551; no voters:
I20260619 11:04:59.021598 17681 raft_consensus.cc:2781] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:04:59.021706 17681 raft_consensus.cc:482] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:04:59.021763 17681 raft_consensus.cc:3037] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:59.022758 17681 raft_consensus.cc:504] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:59.023083 17681 leader_election.cc:288] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [CANDIDATE]: Term 1 election: Requested vote from peers e4784fe2a1dd449482822ddd750a5551 (127.16.61.1:33137), 28f18b480560439e9d8ec8783e37518c (127.16.61.2:43241)
I20260619 11:04:59.023437 17477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159789cc6662409b956d7d6736240b57" candidate_uuid: "2f3849e9676d4180a87eb28baf6f76f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f18b480560439e9d8ec8783e37518c"
I20260619 11:04:59.023528 17343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159789cc6662409b956d7d6736240b57" candidate_uuid: "2f3849e9676d4180a87eb28baf6f76f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4784fe2a1dd449482822ddd750a5551"
I20260619 11:04:59.023543 17477 raft_consensus.cc:3037] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:59.023649 17343 raft_consensus.cc:3037] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:04:59.024732 17343 raft_consensus.cc:2454] T 159789cc6662409b956d7d6736240b57 P e4784fe2a1dd449482822ddd750a5551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3849e9676d4180a87eb28baf6f76f8 in term 1.
I20260619 11:04:59.024892 17477 raft_consensus.cc:2454] T 159789cc6662409b956d7d6736240b57 P 28f18b480560439e9d8ec8783e37518c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3849e9676d4180a87eb28baf6f76f8 in term 1.
I20260619 11:04:59.025022 17545 leader_election.cc:302] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3849e9676d4180a87eb28baf6f76f8, e4784fe2a1dd449482822ddd750a5551; no voters:
I20260619 11:04:59.025202 17681 raft_consensus.cc:2781] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:04:59.025432 17681 raft_consensus.cc:686] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [term 1 LEADER]: Becoming Leader. State: Replica: 2f3849e9676d4180a87eb28baf6f76f8, State: Running, Role: LEADER
I20260619 11:04:59.025753 17681 consensus_queue.cc:237] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } }
I20260619 11:04:59.028640 17197 catalog_manager.cc:5495] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3849e9676d4180a87eb28baf6f76f8 (127.16.61.3). New cstate: current_term: 1 leader_uuid: "2f3849e9676d4180a87eb28baf6f76f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4784fe2a1dd449482822ddd750a5551" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28f18b480560439e9d8ec8783e37518c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3849e9676d4180a87eb28baf6f76f8" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40591 } health_report { overall_health: HEALTHY } } }
Jun 19 11:04:59 dist-test-slave-8vcz krb5kdc[17153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:04:59.064208 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:59.055205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.16.61.1:33137)
0619 11:04:59.055369 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:59.055375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:59.055539 (+ 164us) server_negotiation.cc:407] Connection header received
0619 11:04:59.055706 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:59.055712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:59.055763 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:59.055863 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:59.056535 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.057032 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.057864 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.058082 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.058189 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:59.060660 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:59.060675 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:59.060693 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:59.060714 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:59.062405 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:59.062863 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:59.062872 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:59.062878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:59.062964 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:59.063248 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:59.063253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:59.063255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:59.063491 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:59.063747 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:59.063905 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 11:04:59.064022 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 11:04:59 dist-test-slave-8vcz krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867099, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:04:59 dist-test-slave-8vcz krb5kdc[17153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867099, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:04:59.089607 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:59.081137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.16.61.1:33137)
0619 11:04:59.081290 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:59.081294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:59.081435 (+ 141us) server_negotiation.cc:407] Connection header received
0619 11:04:59.081601 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:59.081604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:59.081651 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:59.081738 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:59.082402 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.083249 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.084150 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.084393 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.084499 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:59.086716 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:59.086720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:59.086723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:59.086754 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:59.088023 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:59.088422 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:59.088426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:59.088428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:59.088505 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:59.088771 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:59.088776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:59.088778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:59.088936 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:59.089066 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:59.089330 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 11:04:59.089445 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 11:04:59 dist-test-slave-8vcz krb5kdc[17153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:04:59 dist-test-slave-8vcz krb5kdc[17153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:04:59.113471 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:04:59.105246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.16.61.1:33137)
0619 11:04:59.105400 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:04:59.105403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:04:59.105465 (+ 62us) server_negotiation.cc:407] Connection header received
0619 11:04:59.105556 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:04:59.105558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:04:59.105593 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:04:59.105657 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:04:59.106386 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.106843 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.107945 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:04:59.108138 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:04:59.108213 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:04:59.110446 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:04:59.110449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:04:59.110451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:04:59.110471 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:04:59.111964 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:59.112372 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:59.112375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:59.112376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:59.112440 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:04:59.112684 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:04:59.112686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:04:59.112687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:04:59.112879 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:04:59.112976 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:04:59.113172 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 11:04:59.113256 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 11:04:59.113839 17658 tablet.cc:2369] T 159789cc6662409b956d7d6736240b57 P 2f3849e9676d4180a87eb28baf6f76f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:04:59.114856 17303 tablet_service.cc:2908] Rejecting scan request for tablet 159789cc6662409b956d7d6736240b57: Uninitialized: clean time has not yet been initialized
I20260619 11:04:59.116813 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 17260
I20260619 11:04:59.122555 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 17394
I20260619 11:04:59.128036 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 17528
I20260619 11:04:59.133270 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 17166
2026-06-19T11:04:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2310 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:04:59 dist-test-slave-8vcz krb5kdc[17698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:04:59 dist-test-slave-8vcz krb5kdc[17698](info): set up 2 sockets
Jun 19 11:04:59 dist-test-slave-8vcz krb5kdc[17698](info): commencing operation
krb5kdc: starting...
W20260619 11:05:00.171766 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:00 dist-test-slave-8vcz krb5kdc[17698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:00.288779 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:40143
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:00.400803 17711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:00.401058 17711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:00.401098 17711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:00.405136 17711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:00.405185 17711 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:05:00.405202 17711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:00.405221 17711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:00.405237 17711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:00.405289 17711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:00.410429 17711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:40143
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:00.411382 17711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:00.412377 17711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:00.420619 17717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:00.420634 17716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:00.420646 17719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:00.420910 17711 server_base.cc:1034] running on GCE node
I20260619 11:05:00.421567 17711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:00.422618 17711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:00.423800 17711 hybrid_clock.cc:648] HybridClock initialized: now 1781867100423787 us; error 26 us; skew 500 ppm
Jun 19 11:05:00 dist-test-slave-8vcz krb5kdc[17698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867100, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:00.429883 17711 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:00.430733 17711 webserver.cc:460] Webserver started at http://127.16.61.62:44171/ using document root <none> and password file <none>
I20260619 11:05:00.431252 17711 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:00.431313 17711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:00.431475 17711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:00.433513 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "e3b4b9b1baac4c7895c1001aae58cb54"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "c5494d97c26701523028c3c1593b2e9f"
server_key_iv: "380e88f07ae594cabddd5ab430045f42"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.434027 17711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "e3b4b9b1baac4c7895c1001aae58cb54"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "c5494d97c26701523028c3c1593b2e9f"
server_key_iv: "380e88f07ae594cabddd5ab430045f42"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.437686 17711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:00.440086 17726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:00.441291 17711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:00.441422 17711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "e3b4b9b1baac4c7895c1001aae58cb54"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "c5494d97c26701523028c3c1593b2e9f"
server_key_iv: "380e88f07ae594cabddd5ab430045f42"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.441525 17711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:00.455407 17711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:00.458124 17711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:00.458312 17711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:00.465718 17711 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:40143
I20260619 11:05:00.465739 17788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:40143 every 8 connection(s)
I20260619 11:05:00.466600 17711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:00.469225 17789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:00.474335 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 17711
I20260619 11:05:00.474431 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:00.474448 17789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54: Bootstrap starting.
I20260619 11:05:00.474694 16628 external_mini_cluster.cc:1428] Setting key ef6367bde84d2b781a02e9eb731104b5
I20260619 11:05:00.476943 17789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:00.477689 17789 log.cc:824] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:00.479496 17789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54: No bootstrap required, opened a new log
Jun 19 11:05:00 dist-test-slave-8vcz krb5kdc[17698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:00.483785 17789 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b4b9b1baac4c7895c1001aae58cb54" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40143 } }
I20260619 11:05:00.484005 17789 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:00.484089 17789 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b4b9b1baac4c7895c1001aae58cb54, State: Initialized, Role: FOLLOWER
I20260619 11:05:00.484571 17789 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b4b9b1baac4c7895c1001aae58cb54" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40143 } }
I20260619 11:05:00.484731 17789 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:00.484824 17789 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:00.484922 17789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:00.485816 17789 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b4b9b1baac4c7895c1001aae58cb54" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40143 } }
I20260619 11:05:00.486155 17789 leader_election.cc:302] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3b4b9b1baac4c7895c1001aae58cb54; no voters:
I20260619 11:05:00.486477 17789 leader_election.cc:288] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:00.486541 17794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:00.486740 17794 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [term 1 LEADER]: Becoming Leader. State: Replica: e3b4b9b1baac4c7895c1001aae58cb54, State: Running, Role: LEADER
I20260619 11:05:00.487021 17794 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b4b9b1baac4c7895c1001aae58cb54" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40143 } }
I20260619 11:05:00.487739 17789 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:00.488593 17796 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b4b9b1baac4c7895c1001aae58cb54. Latest consensus state: current_term: 1 leader_uuid: "e3b4b9b1baac4c7895c1001aae58cb54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b4b9b1baac4c7895c1001aae58cb54" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40143 } } }
I20260619 11:05:00.488739 17796 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:00.489048 17795 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3b4b9b1baac4c7895c1001aae58cb54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b4b9b1baac4c7895c1001aae58cb54" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40143 } } }
I20260619 11:05:00.489138 17795 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:00.490128 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.475961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.16.61.62:40143)
0619 11:05:00.476379 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:00.476395 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:00.476428 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:05:00.477034 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:00.477076 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:00.477411 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:00.477677 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:00.478524 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.479466 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.480595 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.481038 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.481161 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.483912 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:00.483917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:00.483929 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:00.483954 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:00.485964 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.486435 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.486441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.486447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.486597 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.486946 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.486951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.486954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.487403 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:00.487562 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:00.488100 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 11:05:00.488386 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":139,"threads_started":1}
I20260619 11:05:00.490231 17803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:00.490263 17810 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:05:00.493183 17803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:00.498167 17803 catalog_manager.cc:1347] Generated new cluster ID: 7f8f07d774294b3c808beb29b29d0bc9
I20260619 11:05:00.498227 17803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:00.518513 17803 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:00.519745 17803 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:00.524649 17803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54: Generated new TSK 0
I20260619 11:05:00.525355 17803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:00.552969 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40143
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:00.658598 17815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:00.658830 17815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:00.658903 17815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:00.662803 17815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:00.662855 17815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:00.662892 17815 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:00.662958 17815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:00.667989 17815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:00.668946 17815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:00.669924 17815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:00.678963 17820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:00.678949 17823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:00.678949 17821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:00.679178 17815 server_base.cc:1034] running on GCE node
I20260619 11:05:00.679569 17815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:00.680188 17815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:00.681345 17815 hybrid_clock.cc:648] HybridClock initialized: now 1781867100681329 us; error 29 us; skew 500 ppm
Jun 19 11:05:00 dist-test-slave-8vcz krb5kdc[17698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867100, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:00.683974 17815 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:00.684808 17815 webserver.cc:460] Webserver started at http://127.16.61.1:41169/ using document root <none> and password file <none>
I20260619 11:05:00.685328 17815 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:00.685426 17815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:00.685645 17815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:00.687695 17815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "bf9cebfc5fe0406e8fc10aa320fd812b"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "8810ebdf1beafc2a9c8f542530912573"
server_key_iv: "71e20c90d3e77a74750be0dfe357b3c1"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.688315 17815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "bf9cebfc5fe0406e8fc10aa320fd812b"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "8810ebdf1beafc2a9c8f542530912573"
server_key_iv: "71e20c90d3e77a74750be0dfe357b3c1"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.692216 17815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:05:00.694458 17830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:00.695595 17815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:00.695740 17815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "bf9cebfc5fe0406e8fc10aa320fd812b"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "8810ebdf1beafc2a9c8f542530912573"
server_key_iv: "71e20c90d3e77a74750be0dfe357b3c1"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.695852 17815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:00.713043 17815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:00.715799 17815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:00.715991 17815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:00.717340 17815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:00.717417 17815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:00.717506 17815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:00.717553 17815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:05:00 dist-test-slave-8vcz krb5kdc[17698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867100, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:00.727878 17815 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:36051
I20260619 11:05:00.728017 17945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:36051 every 8 connection(s)
I20260619 11:05:00.728940 17815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:00.731842 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.718194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:48735 (local address 127.16.61.62:40143)
0619 11:05:00.718360 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:00.718365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:00.719270 (+ 905us) server_negotiation.cc:407] Connection header received
0619 11:05:00.720456 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:00.720459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:00.720518 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:00.720632 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:00.722077 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.722964 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.724838 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.725187 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.725281 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.728310 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:00.728314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:00.728317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:00.728343 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:00.729969 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.730464 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.730466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.730468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.730541 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.730848 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.730850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.730851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.731016 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:00.731088 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:00.731556 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 11:05:00.731647 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 11:05:00.732623 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.718645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40143 (local address 127.16.61.1:48735)
0619 11:05:00.719102 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:00.719153 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:00.720235 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:00.720810 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:00.720817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:00.721286 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:00.721819 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:00.721835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.723134 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.723139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:00.724707 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:00.724718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.725363 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.725369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:00.725578 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.726244 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:00.726264 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:00.728133 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:00.730079 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:00.730084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:00.730094 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:00.730361 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:00.730620 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:00.730622 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:00.730623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:00.730748 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:00.731138 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:00.731143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:00.731350 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 11:05:00.731499 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 11:05:00.731814 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":149,"threads_started":1}
I20260619 11:05:00.733836 17946 heartbeater.cc:347] Connected to a master server at 127.16.61.62:40143
I20260619 11:05:00.733970 17946 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:00.734484 17946 heartbeater.cc:511] Master 127.16.61.62:40143 requested a full tablet report, sending...
I20260619 11:05:00.736127 17742 ts_manager.cc:194] Registered new tserver with Master: bf9cebfc5fe0406e8fc10aa320fd812b (127.16.61.1:36051)
I20260619 11:05:00.737442 17742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:48735
I20260619 11:05:00.738662 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 17815
I20260619 11:05:00.738770 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:00.739065 16628 external_mini_cluster.cc:1428] Setting key a23ac1f531c0d600b6a57e0f1abb0f59
I20260619 11:05:00.743435 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.736091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:48853 (local address 127.16.61.62:40143)
0619 11:05:00.736273 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:00.736277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:00.736328 (+ 51us) server_negotiation.cc:407] Connection header received
0619 11:05:00.736449 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:00.736451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:00.736486 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:00.736548 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:00.737011 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.737475 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.738507 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.738830 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.738952 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.740254 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:00.740257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:00.740259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:00.740278 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:00.741887 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.742323 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.742329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.742333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.742426 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.742738 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.742741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.742742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.742886 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:00.742967 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:00.743250 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 11:05:00.743333 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 11:05:00.743590 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.735971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40143 (local address 127.16.61.1:48853)
0619 11:05:00.736257 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:00.736272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:00.736368 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:00.736589 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:00.736592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:00.736800 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:00.736906 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:00.736913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.737621 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.737625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:00.738356 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:00.738365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.738947 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.738951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:00.739034 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.739712 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:00.739729 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:00.740144 (+ 415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:00.742007 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:00.742009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:00.742011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:00.742191 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:00.742531 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:00.742535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:00.742538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:00.742645 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:00.742986 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:00.742990 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:00.743092 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 11:05:00.743203 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:05:00.743358 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:00.765987 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40143
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:00.874989 17955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:00.875252 17955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:00.875294 17955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:00.879343 17955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:00.879400 17955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:00.879437 17955 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:00.879469 17955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:00.884608 17955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:00.885600 17955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:00.886561 17955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:00.895816 17963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:00.895792 17960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:00.895841 17961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:00.896112 17955 server_base.cc:1034] running on GCE node
I20260619 11:05:00.896493 17955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:00.897120 17955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:00.898326 17955 hybrid_clock.cc:648] HybridClock initialized: now 1781867100898314 us; error 29 us; skew 500 ppm
Jun 19 11:05:00 dist-test-slave-8vcz krb5kdc[17698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867100, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:00.901119 17955 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:00.901973 17955 webserver.cc:460] Webserver started at http://127.16.61.2:32805/ using document root <none> and password file <none>
I20260619 11:05:00.902505 17955 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:00.902590 17955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:00.902807 17955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:00.904814 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "b4799d497f514e1d81d7b364b0d86876"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "a8e1469d29e3b59f1315f8ef1ce18c2c"
server_key_iv: "fc2032db5ca157994c209f1867a7ac88"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.905376 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "b4799d497f514e1d81d7b364b0d86876"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "a8e1469d29e3b59f1315f8ef1ce18c2c"
server_key_iv: "fc2032db5ca157994c209f1867a7ac88"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.908844 17955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:05:00.911125 17970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:00.912420 17955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:05:00.912552 17955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "b4799d497f514e1d81d7b364b0d86876"
format_stamp: "Formatted at 2026-06-19 11:05:00 on dist-test-slave-8vcz"
server_key: "a8e1469d29e3b59f1315f8ef1ce18c2c"
server_key_iv: "fc2032db5ca157994c209f1867a7ac88"
server_key_version: "encryptionkey@0"
I20260619 11:05:00.912657 17955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:00.939178 17955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:00.942490 17955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:00.942682 17955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:00.944144 17955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:00.944209 17955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:00.944298 17955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:00.944342 17955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:05:00 dist-test-slave-8vcz krb5kdc[17698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867100, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:00.955366 17955 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:46107
I20260619 11:05:00.955437 18085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:46107 every 8 connection(s)
I20260619 11:05:00.956434 17955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:00.957715 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.944780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:47983 (local address 127.16.61.62:40143)
0619 11:05:00.944942 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:00.944948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:00.946307 (+ 1359us) server_negotiation.cc:407] Connection header received
0619 11:05:00.947328 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:00.947332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:00.947395 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:00.947501 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:00.948659 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.949241 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.950412 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.950628 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.950740 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.953912 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:00.953919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:00.953923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:00.953955 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:00.955659 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.956154 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.956158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.956160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.956248 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.956579 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.956583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.956584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.956769 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:00.956872 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:00.957436 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 11:05:00.957551 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:05:00.958425 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.945678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40143 (local address 127.16.61.2:47983)
0619 11:05:00.946152 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:00.946199 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:00.947131 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:00.947745 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:00.947756 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:00.948075 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:00.948516 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:00.948529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.949374 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.949377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:00.950284 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:00.950294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.950824 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.950833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:00.951072 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.951841 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:00.951861 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:00.953751 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:00.955796 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:00.955801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:00.955811 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:00.956050 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:00.956341 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:00.956343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:00.956344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:00.956465 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:00.956881 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:00.956890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:00.957164 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 11:05:00.957382 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 11:05:00.957670 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260619 11:05:00.959780 18086 heartbeater.cc:347] Connected to a master server at 127.16.61.62:40143
I20260619 11:05:00.959932 18086 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:00.960484 18086 heartbeater.cc:511] Master 127.16.61.62:40143 requested a full tablet report, sending...
I20260619 11:05:00.961555 17742 ts_manager.cc:194] Registered new tserver with Master: b4799d497f514e1d81d7b364b0d86876 (127.16.61.2:46107)
I20260619 11:05:00.961580 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 17955
I20260619 11:05:00.961683 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:00.962019 16628 external_mini_cluster.cc:1428] Setting key 82cb6cb703c99fb5393fd2c536cba606
I20260619 11:05:00.962311 17742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:47983
I20260619 11:05:00.970453 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.962090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:55189 (local address 127.16.61.62:40143)
0619 11:05:00.962196 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:00.962200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:00.962415 (+ 215us) server_negotiation.cc:407] Connection header received
0619 11:05:00.962716 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:00.962719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:00.962761 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:00.962858 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:00.963321 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.964055 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.965352 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.965725 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.965849 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.967003 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:00.967006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:00.967008 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:00.967028 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:00.968676 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.969212 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.969215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.969216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.969308 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:00.969637 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:00.969640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:00.969641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:00.969836 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:00.969916 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:00.970239 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 11:05:00.970343 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:05:00.970558 18092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:00.961991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40143 (local address 127.16.61.2:55189)
0619 11:05:00.962335 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:00.962351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:00.962485 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:00.962873 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:00.962877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:00.963079 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:00.963186 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:00.963192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.964207 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.964213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:00.965201 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:00.965215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:00.965845 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:00.965848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:00.965913 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:00.966405 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:00.966424 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:00.966866 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:00.968806 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:00.968812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:00.968815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:00.969087 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:00.969416 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:00.969421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:00.969424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:00.969532 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:00.969947 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:00.969952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:00.970053 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 11:05:00.970175 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 11:05:00.970334 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:05:00.990499 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40143
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:39637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:01.100154 18095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:01.100386 18095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:01.100467 18095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:01.104420 18095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:01.104468 18095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:01.104538 18095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:01.104625 18095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:01.109699 18095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:01.110638 18095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:01.111588 18095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:01.120882 18100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:01.120935 18103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:01.120962 18101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:01.121157 18095 server_base.cc:1034] running on GCE node
I20260619 11:05:01.121600 18095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:01.122196 18095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:01.123376 18095 hybrid_clock.cc:648] HybridClock initialized: now 1781867101123348 us; error 39 us; skew 500 ppm
Jun 19 11:05:01 dist-test-slave-8vcz krb5kdc[17698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867101, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:01.126077 18095 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:01.126919 18095 webserver.cc:460] Webserver started at http://127.16.61.3:44703/ using document root <none> and password file <none>
I20260619 11:05:01.127483 18095 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:01.127580 18095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:01.127844 18095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:01.129776 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "a647dd379d354afa8a2e21c5a25b46e4"
format_stamp: "Formatted at 2026-06-19 11:05:01 on dist-test-slave-8vcz"
server_key: "1160c3a9ff2d0b97978f738a372e9da8"
server_key_iv: "47e770e1599e3bb76c746af08f6d394d"
server_key_version: "encryptionkey@0"
I20260619 11:05:01.130337 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "a647dd379d354afa8a2e21c5a25b46e4"
format_stamp: "Formatted at 2026-06-19 11:05:01 on dist-test-slave-8vcz"
server_key: "1160c3a9ff2d0b97978f738a372e9da8"
server_key_iv: "47e770e1599e3bb76c746af08f6d394d"
server_key_version: "encryptionkey@0"
I20260619 11:05:01.133977 18095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:01.136350 18110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:01.137610 18095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:01.137766 18095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "a647dd379d354afa8a2e21c5a25b46e4"
format_stamp: "Formatted at 2026-06-19 11:05:01 on dist-test-slave-8vcz"
server_key: "1160c3a9ff2d0b97978f738a372e9da8"
server_key_iv: "47e770e1599e3bb76c746af08f6d394d"
server_key_version: "encryptionkey@0"
I20260619 11:05:01.137884 18095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:01.152179 18095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:01.154881 18095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:01.155082 18095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:01.156577 18095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:01.156656 18095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:01.156721 18095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:01.156774 18095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:05:01 dist-test-slave-8vcz krb5kdc[17698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867101, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:01.167528 18095 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:37837
I20260619 11:05:01.167639 18225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:37837 every 8 connection(s)
I20260619 11:05:01.168609 18095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:01.170182 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.157142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:37741 (local address 127.16.61.62:40143)
0619 11:05:01.157288 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.157293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.158542 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 11:05:01.159555 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.159558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.159614 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.159735 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:01.161410 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.162295 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.163611 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.163893 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.164013 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.166471 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:01.166475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:01.166478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:01.166505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:01.168228 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:01.168700 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:01.168704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:01.168706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:01.168806 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:01.169187 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:01.169190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:01.169192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:01.169415 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:01.169509 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.169977 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.170068 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:05:01.170951 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.157929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40143 (local address 127.16.61.3:37741)
0619 11:05:01.158366 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.158413 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.159346 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.159985 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.159994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.160485 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:01.161224 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.161248 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.162418 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.162422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.163484 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.163493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.164055 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.164059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.164218 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.164784 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:01.164798 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:01.166291 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:01.168337 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:01.168343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:01.168353 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:01.168590 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:01.168915 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:01.168917 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:01.168918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:01.169039 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:01.169542 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:01.169549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:01.169774 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.169926 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.170142 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":129,"threads_started":1}
I20260619 11:05:01.172139 18226 heartbeater.cc:347] Connected to a master server at 127.16.61.62:40143
I20260619 11:05:01.172271 18226 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:01.172750 18226 heartbeater.cc:511] Master 127.16.61.62:40143 requested a full tablet report, sending...
I20260619 11:05:01.173804 17743 ts_manager.cc:194] Registered new tserver with Master: a647dd379d354afa8a2e21c5a25b46e4 (127.16.61.3:37837)
I20260619 11:05:01.174573 17743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:37741
I20260619 11:05:01.176522 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 18095
I20260619 11:05:01.176645 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:01.176903 16628 external_mini_cluster.cc:1428] Setting key 3b4ae983d50721bdbda559a01d04b782
I20260619 11:05:01.179020 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:05:01.181679 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.174186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40143 (local address 127.16.61.3:45887)
0619 11:05:01.174497 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.174509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.174627 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.174922 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.174924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.175130 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:01.175233 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.175239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.176205 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.176207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.176874 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.176882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.177412 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.177415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.177516 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.178031 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:01.178048 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:01.178448 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:01.180354 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:01.180356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:01.180358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:01.180539 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:01.180827 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:01.180829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:01.180830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:01.180899 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:01.181310 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:01.181312 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:01.181378 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.181456 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.181558 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260619 11:05:01.181854 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.174262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:45887 (local address 127.16.61.62:40143)
0619 11:05:01.174377 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.174380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.174584 (+ 204us) server_negotiation.cc:407] Connection header received
0619 11:05:01.174722 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.174725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.174769 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.174858 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:01.175355 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.176084 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.176994 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.177271 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.177396 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.178569 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:01.178572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:01.178574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:01.178593 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:01.180245 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:01.180639 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:01.180645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:01.180650 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:01.180735 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:01.180994 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:01.180999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:01.181002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:01.181218 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:01.181306 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.181548 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.181666 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 11:05:01 dist-test-slave-8vcz krb5kdc[17698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:05:01 dist-test-slave-8vcz krb5kdc[17698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:01.205518 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.197474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.16.61.62:40143)
0619 11:05:01.197593 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.197597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.197608 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:05:01.197643 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.197645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.197679 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.197739 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:01.198398 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.198911 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.199844 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.200054 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.200146 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.202417 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:01.202421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:01.202423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:01.202444 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:01.203999 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:01.204385 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:01.204388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:01.204389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:01.204460 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:01.204682 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:01.204685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:01.204686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:01.204851 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:01.204946 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.205201 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.205341 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:05:01.208432 17743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:01.211371 17743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:01.225956 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.220861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.16.61.3:37837)
0619 11:05:01.221225 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.221236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.221255 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:05:01.221324 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.221343 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.221525 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.221699 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.221916 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.222651 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.224641 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.225262 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.225403 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.225573 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.225703 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.225790 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":102,"threads_started":1}
I20260619 11:05:01.225941 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.220483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:37837 (local address 127.0.0.1:43270)
0619 11:05:01.221032 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.221045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.221127 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.221677 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.221681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.221699 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.221787 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.221796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.222776 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.222781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.224493 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.224514 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.225411 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.225415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.225516 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.225553 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.225665 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.225786 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":88,"thread_start_us":85,"threads_started":1}
I20260619 11:05:01.226116 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.219901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:36051 (local address 127.0.0.1:59256)
0619 11:05:01.220698 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.220713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.220784 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.221429 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.221436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.221459 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.221590 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.221597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.223064 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.223068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.224419 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.224428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.225660 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.225667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.225775 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.225822 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.225886 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.225964 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":112,"thread_start_us":81,"threads_started":1}
I20260619 11:05:01.226617 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.220524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.16.61.1:36051)
0619 11:05:01.220905 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.220920 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.220954 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:05:01.221035 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.221040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.221252 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.221411 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.221732 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.222919 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.224557 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.225330 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.225439 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.225593 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.226388 (+ 795us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.226481 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":75,"threads_started":1}
I20260619 11:05:01.226116 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.219745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:46107 (local address 127.0.0.1:44494)
0619 11:05:01.220424 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.220472 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.220585 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.221810 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.221814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.221831 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.221906 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.221912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.223473 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.223476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.224716 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.224724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.225765 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.225768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.225873 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.225897 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.225938 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.225985 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":9,"thread_start_us":77,"threads_started":1}
I20260619 11:05:01.227644 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.219891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.16.61.2:46107)
0619 11:05:01.221374 (+ 1483us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.221386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.221408 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:05:01.221475 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.221479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.221669 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.222040 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.222160 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.223299 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.224841 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.225629 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.227165 (+ 1536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.227363 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.227417 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.227506 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1361,"thread_start_us":100,"threads_started":1}
I20260619 11:05:01.229223 17880 tablet_service.cc:1467] Processing CreateTablet for tablet ca691dde39bd424c98a91585e8ff877f (DEFAULT_TABLE table=test-table [id=84061a41bf77409ab5713bd93c50252c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:01.229350 18157 tablet_service.cc:1467] Processing CreateTablet for tablet ca691dde39bd424c98a91585e8ff877f (DEFAULT_TABLE table=test-table [id=84061a41bf77409ab5713bd93c50252c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:01.230480 18157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca691dde39bd424c98a91585e8ff877f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.230746 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca691dde39bd424c98a91585e8ff877f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.230381 18020 tablet_service.cc:1467] Processing CreateTablet for tablet ca691dde39bd424c98a91585e8ff877f (DEFAULT_TABLE table=test-table [id=84061a41bf77409ab5713bd93c50252c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:01.231340 18020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca691dde39bd424c98a91585e8ff877f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.237270 18249 tablet_bootstrap.cc:492] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: Bootstrap starting.
I20260619 11:05:01.239037 18251 tablet_bootstrap.cc:492] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: Bootstrap starting.
I20260619 11:05:01.239037 18250 tablet_bootstrap.cc:492] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: Bootstrap starting.
I20260619 11:05:01.240025 18249 tablet_bootstrap.cc:654] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.240757 18249 log.cc:824] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:01.241123 18251 tablet_bootstrap.cc:654] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.241123 18250 tablet_bootstrap.cc:654] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.241937 18251 log.cc:824] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:01.241983 18250 log.cc:824] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:01.242930 18249 tablet_bootstrap.cc:492] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: No bootstrap required, opened a new log
I20260619 11:05:01.243116 18249 ts_tablet_manager.cc:1397] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:05:01.243657 18251 tablet_bootstrap.cc:492] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: No bootstrap required, opened a new log
I20260619 11:05:01.243853 18250 tablet_bootstrap.cc:492] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: No bootstrap required, opened a new log
I20260619 11:05:01.243945 18251 ts_tablet_manager.cc:1397] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:05:01.244041 18250 ts_tablet_manager.cc:1397] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:05:01.246912 18251 raft_consensus.cc:348] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.247146 18251 raft_consensus.cc:374] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.247225 18251 raft_consensus.cc:729] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4799d497f514e1d81d7b364b0d86876, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.247102 18249 raft_consensus.cc:348] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.247375 18249 raft_consensus.cc:374] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.247439 18249 raft_consensus.cc:729] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9cebfc5fe0406e8fc10aa320fd812b, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.247771 18251 consensus_queue.cc:260] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.247715 18250 raft_consensus.cc:348] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.247859 18249 consensus_queue.cc:260] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.247946 18250 raft_consensus.cc:374] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.247989 18250 raft_consensus.cc:729] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a647dd379d354afa8a2e21c5a25b46e4, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.248396 18086 heartbeater.cc:503] Master 127.16.61.62:40143 was elected leader, sending a full tablet report...
I20260619 11:05:01.248740 18249 ts_tablet_manager.cc:1428] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:05:01.248750 18251 ts_tablet_manager.cc:1428] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:05:01.248680 18250 consensus_queue.cc:260] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.249042 17946 heartbeater.cc:503] Master 127.16.61.62:40143 was elected leader, sending a full tablet report...
I20260619 11:05:01.249501 18250 ts_tablet_manager.cc:1428] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:05:01.250468 18226 heartbeater.cc:503] Master 127.16.61.62:40143 was elected leader, sending a full tablet report...
I20260619 11:05:01.282208 18256 raft_consensus.cc:482] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:01.282411 18256 raft_consensus.cc:504] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.283496 18256 leader_election.cc:288] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4799d497f514e1d81d7b364b0d86876 (127.16.61.2:46107), a647dd379d354afa8a2e21c5a25b46e4 (127.16.61.3:37837)
I20260619 11:05:01.288988 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.284246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:57097 (local address 127.16.61.3:37837)
0619 11:05:01.284364 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.284368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.284521 (+ 153us) server_negotiation.cc:407] Connection header received
0619 11:05:01.284604 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.284606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.284674 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.284754 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.284974 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.285699 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.287515 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.288155 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.288270 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.288332 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.288782 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.288859 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:01.288988 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.283979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:46115 (local address 127.16.61.2:46107)
0619 11:05:01.284109 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.284114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.284129 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:05:01.284183 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.284186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.284236 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.284321 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.284692 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.285511 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.287515 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.288178 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.288269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.288332 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.288778 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.288859 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:05:01.289060 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.283706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:46107 (local address 127.16.61.1:46115)
0619 11:05:01.283970 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.283985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.284082 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.284414 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.284417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.284439 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.284573 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.284583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.285712 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.285715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.287361 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.287373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.288394 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.288398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.288568 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.288637 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.288720 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.288820 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":180,"thread_start_us":123,"threads_started":1}
I20260619 11:05:01.289063 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.284123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:37837 (local address 127.16.61.1:57097)
0619 11:05:01.284427 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.284439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.284527 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.284771 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.284773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.284788 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.284866 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.284870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.285825 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.285828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.287363 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.287373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.288394 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.288398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.288612 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.288648 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.288720 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.288820 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":210,"thread_start_us":130,"threads_started":1}
I20260619 11:05:01.289796 18040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca691dde39bd424c98a91585e8ff877f" candidate_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4799d497f514e1d81d7b364b0d86876" is_pre_election: true
I20260619 11:05:01.289819 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca691dde39bd424c98a91585e8ff877f" candidate_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a647dd379d354afa8a2e21c5a25b46e4" is_pre_election: true
I20260619 11:05:01.290131 18040 raft_consensus.cc:2454] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9cebfc5fe0406e8fc10aa320fd812b in term 0.
I20260619 11:05:01.290130 18180 raft_consensus.cc:2454] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9cebfc5fe0406e8fc10aa320fd812b in term 0.
I20260619 11:05:01.290613 17831 leader_election.cc:302] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a647dd379d354afa8a2e21c5a25b46e4, bf9cebfc5fe0406e8fc10aa320fd812b; no voters:
I20260619 11:05:01.290899 18256 raft_consensus.cc:2781] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:01.290992 18256 raft_consensus.cc:482] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:01.291035 18256 raft_consensus.cc:3037] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.292135 18256 raft_consensus.cc:504] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.292474 18256 leader_election.cc:288] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [CANDIDATE]: Term 1 election: Requested vote from peers b4799d497f514e1d81d7b364b0d86876 (127.16.61.2:46107), a647dd379d354afa8a2e21c5a25b46e4 (127.16.61.3:37837)
I20260619 11:05:01.292861 18040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca691dde39bd424c98a91585e8ff877f" candidate_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4799d497f514e1d81d7b364b0d86876"
I20260619 11:05:01.292876 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca691dde39bd424c98a91585e8ff877f" candidate_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a647dd379d354afa8a2e21c5a25b46e4"
I20260619 11:05:01.292961 18040 raft_consensus.cc:3037] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.293007 18180 raft_consensus.cc:3037] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.293980 18040 raft_consensus.cc:2454] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9cebfc5fe0406e8fc10aa320fd812b in term 1.
I20260619 11:05:01.294111 18180 raft_consensus.cc:2454] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9cebfc5fe0406e8fc10aa320fd812b in term 1.
I20260619 11:05:01.294346 17831 leader_election.cc:302] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4799d497f514e1d81d7b364b0d86876, bf9cebfc5fe0406e8fc10aa320fd812b; no voters:
I20260619 11:05:01.294529 18256 raft_consensus.cc:2781] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:01.294768 18256 raft_consensus.cc:686] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 LEADER]: Becoming Leader. State: Replica: bf9cebfc5fe0406e8fc10aa320fd812b, State: Running, Role: LEADER
I20260619 11:05:01.295101 18256 consensus_queue.cc:237] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.299068 17743 catalog_manager.cc:5495] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9cebfc5fe0406e8fc10aa320fd812b (127.16.61.1). New cstate: current_term: 1 leader_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:01.330691 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.327089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.16.61.1:36051)
0619 11:05:01.327281 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.327284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.327299 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:05:01.327425 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.327427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.327481 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.327568 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:01.327891 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.328403 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.329299 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.329497 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.329570 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.329887 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:01.330296 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:01.330381 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.330534 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.330582 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 11:05:01.336890 18040 raft_consensus.cc:1261] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Refusing update from remote peer bf9cebfc5fe0406e8fc10aa320fd812b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:01.337044 18180 raft_consensus.cc:1261] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Refusing update from remote peer bf9cebfc5fe0406e8fc10aa320fd812b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:01.337834 18256 consensus_queue.cc:1035] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:01.338089 18260 consensus_queue.cc:1035] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:01.346460 18263 mvcc.cc:205] Tried to move back new op lower bound from 7298527647068127232 to 7298527646908317696. Current Snapshot: MvccSnapshot[applied={T|T < 7298527647068127232}]
I20260619 11:05:01.346657 18264 mvcc.cc:205] Tried to move back new op lower bound from 7298527647068127232 to 7298527646908317696. Current Snapshot: MvccSnapshot[applied={T|T < 7298527647068127232}]
I20260619 11:05:01.347244 18265 mvcc.cc:205] Tried to move back new op lower bound from 7298527647068127232 to 7298527646908317696. Current Snapshot: MvccSnapshot[applied={T|T < 7298527647068127232}]
I20260619 11:05:01.359098 17741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:01.359337 17741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:01.362188 17741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54: Sending DeleteTablet for 3 replicas of tablet ca691dde39bd424c98a91585e8ff877f
I20260619 11:05:01.363077 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet ca691dde39bd424c98a91585e8ff877f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:01 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:59256
I20260619 11:05:01.363096 18157 tablet_service.cc:1514] Processing DeleteTablet for tablet ca691dde39bd424c98a91585e8ff877f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:01 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:43270
I20260619 11:05:01.363137 18020 tablet_service.cc:1514] Processing DeleteTablet for tablet ca691dde39bd424c98a91585e8ff877f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:01 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:44494
I20260619 11:05:01.363641 18273 tablet_replica.cc:330] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: stopping tablet replica
I20260619 11:05:01.363902 18275 tablet_replica.cc:330] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: stopping tablet replica
I20260619 11:05:01.364020 18273 raft_consensus.cc:2229] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:01.364146 18275 raft_consensus.cc:2229] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:01.364378 18273 raft_consensus.cc:2258] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:01.364607 18275 raft_consensus.cc:2258] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:01.364621 17741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:05:01.364993 18274 tablet_replica.cc:330] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: stopping tablet replica
W20260619 11:05:01.365059 17741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:01.365296 18274 raft_consensus.cc:2229] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:01.365700 18274 raft_consensus.cc:2258] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:01.367575 18274 ts_tablet_manager.cc:1905] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:01.368579 18275 ts_tablet_manager.cc:1905] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:01.370013 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 2bbeed03b8294322862b35d4705f0c93 (DEFAULT_TABLE table=test-table [id=79b23f0911294524a8bd9993872eb26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:01.370297 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bbeed03b8294322862b35d4705f0c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.370270 18157 tablet_service.cc:1467] Processing CreateTablet for tablet 2bbeed03b8294322862b35d4705f0c93 (DEFAULT_TABLE table=test-table [id=79b23f0911294524a8bd9993872eb26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:01.370283 18020 tablet_service.cc:1467] Processing CreateTablet for tablet 2bbeed03b8294322862b35d4705f0c93 (DEFAULT_TABLE table=test-table [id=79b23f0911294524a8bd9993872eb26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:01.370517 18157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bbeed03b8294322862b35d4705f0c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.370566 18020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bbeed03b8294322862b35d4705f0c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.372677 18251 tablet_bootstrap.cc:492] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: Bootstrap starting.
I20260619 11:05:01.373139 18249 tablet_bootstrap.cc:492] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: Bootstrap starting.
I20260619 11:05:01.373437 18250 tablet_bootstrap.cc:492] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: Bootstrap starting.
I20260619 11:05:01.373685 18251 tablet_bootstrap.cc:654] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.374109 18249 tablet_bootstrap.cc:654] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.374433 18250 tablet_bootstrap.cc:654] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.375718 18250 tablet_bootstrap.cc:492] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: No bootstrap required, opened a new log
I20260619 11:05:01.375804 18250 ts_tablet_manager.cc:1397] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:01.376232 18250 raft_consensus.cc:348] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.376348 18250 raft_consensus.cc:374] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.376379 18250 raft_consensus.cc:729] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a647dd379d354afa8a2e21c5a25b46e4, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.376494 18250 consensus_queue.cc:260] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.376516 18274 ts_tablet_manager.cc:1918] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:01.376612 18274 log.cc:1196] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/ca691dde39bd424c98a91585e8ff877f
I20260619 11:05:01.376718 18250 ts_tablet_manager.cc:1428] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:01.376952 18274 ts_tablet_manager.cc:1939] T ca691dde39bd424c98a91585e8ff877f P bf9cebfc5fe0406e8fc10aa320fd812b: Deleting consensus metadata
I20260619 11:05:01.378141 17730 catalog_manager.cc:4841] TS bf9cebfc5fe0406e8fc10aa320fd812b (127.16.61.1:36051): tablet ca691dde39bd424c98a91585e8ff877f (table test-table [id=84061a41bf77409ab5713bd93c50252c]) successfully deleted
I20260619 11:05:01.378542 18273 ts_tablet_manager.cc:1905] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:01.378587 18275 ts_tablet_manager.cc:1918] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:01.378675 18275 log.cc:1196] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/ca691dde39bd424c98a91585e8ff877f
I20260619 11:05:01.379029 18275 ts_tablet_manager.cc:1939] T ca691dde39bd424c98a91585e8ff877f P b4799d497f514e1d81d7b364b0d86876: Deleting consensus metadata
I20260619 11:05:01.380347 17727 catalog_manager.cc:4841] TS b4799d497f514e1d81d7b364b0d86876 (127.16.61.2:46107): tablet ca691dde39bd424c98a91585e8ff877f (table test-table [id=84061a41bf77409ab5713bd93c50252c]) successfully deleted
I20260619 11:05:01.380924 18251 tablet_bootstrap.cc:492] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: No bootstrap required, opened a new log
I20260619 11:05:01.381024 18251 ts_tablet_manager.cc:1397] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 11:05:01.381261 18249 tablet_bootstrap.cc:492] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: No bootstrap required, opened a new log
I20260619 11:05:01.381361 18249 ts_tablet_manager.cc:1397] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 11:05:01.381505 18251 raft_consensus.cc:348] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.381621 18251 raft_consensus.cc:374] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.381660 18251 raft_consensus.cc:729] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4799d497f514e1d81d7b364b0d86876, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.381788 18251 consensus_queue.cc:260] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.381822 18249 raft_consensus.cc:348] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.381943 18249 raft_consensus.cc:374] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.381985 18249 raft_consensus.cc:729] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9cebfc5fe0406e8fc10aa320fd812b, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.382090 18251 ts_tablet_manager.cc:1428] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:01.382110 18249 consensus_queue.cc:260] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.382381 18249 ts_tablet_manager.cc:1428] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:01.383220 18273 ts_tablet_manager.cc:1918] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:01.383340 18273 log.cc:1196] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/ca691dde39bd424c98a91585e8ff877f
I20260619 11:05:01.383641 18273 ts_tablet_manager.cc:1939] T ca691dde39bd424c98a91585e8ff877f P a647dd379d354afa8a2e21c5a25b46e4: Deleting consensus metadata
I20260619 11:05:01.384931 17727 catalog_manager.cc:4841] TS a647dd379d354afa8a2e21c5a25b46e4 (127.16.61.3:37837): tablet ca691dde39bd424c98a91585e8ff877f (table test-table [id=84061a41bf77409ab5713bd93c50252c]) successfully deleted
W20260619 11:05:01.419795 18227 tablet.cc:2369] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:01.457593 18087 tablet.cc:2369] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:01.480518 17947 tablet.cc:2369] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:01.777686 18257 raft_consensus.cc:482] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:01.777989 18257 raft_consensus.cc:504] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.779575 18257 leader_election.cc:288] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9cebfc5fe0406e8fc10aa320fd812b (127.16.61.1:36051), b4799d497f514e1d81d7b364b0d86876 (127.16.61.2:46107)
I20260619 11:05:01.783627 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.779948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:45595 (local address 127.16.61.2:46107)
0619 11:05:01.780078 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.780085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.780307 (+ 222us) server_negotiation.cc:407] Connection header received
0619 11:05:01.780438 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.780443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.780498 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.780571 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.780910 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.781522 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.782631 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.783026 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.783105 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.783196 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.783417 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.783493 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:01.783656 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.779968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:50225 (local address 127.16.61.1:36051)
0619 11:05:01.780126 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.780132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.780493 (+ 361us) server_negotiation.cc:407] Connection header received
0619 11:05:01.780604 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.780607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.780650 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.780754 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.781092 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.781561 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.782677 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.783058 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.783189 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.783250 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.783484 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.783556 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:05:01.783650 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.779856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:46107 (local address 127.16.61.3:45595)
0619 11:05:01.780126 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.780142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.780310 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.780602 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.780604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.780624 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.780766 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.780774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.781647 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.781652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.782495 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.782504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.783156 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.783162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.783272 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.783304 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.783365 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.783453 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":3,"thread_start_us":76,"threads_started":1}
I20260619 11:05:01.783653 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.779860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:36051 (local address 127.16.61.3:50225)
0619 11:05:01.780392 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.780407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.780494 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.780800 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.780806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.780881 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.780974 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.780981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.781712 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.781715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.782503 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.782512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.783222 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.783226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.783318 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.783337 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.783441 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.783500 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":306,"thread_start_us":65,"threads_started":1}
I20260619 11:05:01.784125 18040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbeed03b8294322862b35d4705f0c93" candidate_uuid: "a647dd379d354afa8a2e21c5a25b46e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4799d497f514e1d81d7b364b0d86876" is_pre_election: true
I20260619 11:05:01.784319 18040 raft_consensus.cc:2454] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a647dd379d354afa8a2e21c5a25b46e4 in term 0.
I20260619 11:05:01.784538 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbeed03b8294322862b35d4705f0c93" candidate_uuid: "a647dd379d354afa8a2e21c5a25b46e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" is_pre_election: true
I20260619 11:05:01.784749 18111 leader_election.cc:302] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a647dd379d354afa8a2e21c5a25b46e4, b4799d497f514e1d81d7b364b0d86876; no voters:
I20260619 11:05:01.784807 17892 raft_consensus.cc:2454] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a647dd379d354afa8a2e21c5a25b46e4 in term 0.
I20260619 11:05:01.785061 18257 raft_consensus.cc:2781] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:01.785130 18257 raft_consensus.cc:482] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:01.785185 18257 raft_consensus.cc:3037] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.786170 18257 raft_consensus.cc:504] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.786540 18257 leader_election.cc:288] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9cebfc5fe0406e8fc10aa320fd812b (127.16.61.1:36051), b4799d497f514e1d81d7b364b0d86876 (127.16.61.2:46107)
I20260619 11:05:01.786955 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbeed03b8294322862b35d4705f0c93" candidate_uuid: "a647dd379d354afa8a2e21c5a25b46e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b"
I20260619 11:05:01.787012 18040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbeed03b8294322862b35d4705f0c93" candidate_uuid: "a647dd379d354afa8a2e21c5a25b46e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4799d497f514e1d81d7b364b0d86876"
I20260619 11:05:01.787086 17892 raft_consensus.cc:3037] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.787158 18040 raft_consensus.cc:3037] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.788236 18040 raft_consensus.cc:2454] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a647dd379d354afa8a2e21c5a25b46e4 in term 1.
I20260619 11:05:01.788404 17892 raft_consensus.cc:2454] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a647dd379d354afa8a2e21c5a25b46e4 in term 1.
I20260619 11:05:01.788564 18111 leader_election.cc:302] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a647dd379d354afa8a2e21c5a25b46e4, b4799d497f514e1d81d7b364b0d86876; no voters:
I20260619 11:05:01.788777 18257 raft_consensus.cc:2781] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:01.788964 18257 raft_consensus.cc:686] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 1 LEADER]: Becoming Leader. State: Replica: a647dd379d354afa8a2e21c5a25b46e4, State: Running, Role: LEADER
I20260619 11:05:01.789290 18257 consensus_queue.cc:237] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.791579 17743 catalog_manager.cc:5495] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a647dd379d354afa8a2e21c5a25b46e4 (127.16.61.3). New cstate: current_term: 1 leader_uuid: "a647dd379d354afa8a2e21c5a25b46e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } health_report { overall_health: HEALTHY } } }
Jun 19 11:05:01 dist-test-slave-8vcz krb5kdc[17698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867100, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:01.854467 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.845483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40143 (local address 127.0.0.1:44130)
0619 11:05:01.845927 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.845945 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.846112 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.846377 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.846380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.846922 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:01.847061 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.847073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.848040 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.848043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.848616 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.848624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.849059 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.849062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.849126 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.849696 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:01.849707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:01.851188 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:01.852948 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:01.852956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:01.852960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:01.853202 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:01.853505 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:01.853507 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:01.853508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:01.853586 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:01.853928 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:01.853934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:01.854036 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.854130 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.854278 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":127,"threads_started":1}
I20260619 11:05:01.854488 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.845593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.16.61.62:40143)
0619 11:05:01.845927 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.845937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.846049 (+ 112us) server_negotiation.cc:407] Connection header received
0619 11:05:01.846212 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.846216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.846261 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.846363 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:01.847247 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.847924 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.848727 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.848949 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.849028 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.851322 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:01.851325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:01.851327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:01.851347 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:01.852833 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:01.853328 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:01.853331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:01.853332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:01.853422 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:01.853667 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:01.853669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:01.853670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:01.853831 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:01.853910 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.854188 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.854335 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":283,"thread_start_us":85,"threads_started":1}
I20260619 11:05:01.858445 17743 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:44130:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:05:01.859191 17743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:01.864780 18157 tablet_service.cc:1467] Processing CreateTablet for tablet 66eaf1357ed146ae97d0df37665752a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a09307bf490a44f1a581cddd105dfa68]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:01.864712 17880 tablet_service.cc:1467] Processing CreateTablet for tablet 66eaf1357ed146ae97d0df37665752a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a09307bf490a44f1a581cddd105dfa68]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:01.864804 18020 tablet_service.cc:1467] Processing CreateTablet for tablet 66eaf1357ed146ae97d0df37665752a3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a09307bf490a44f1a581cddd105dfa68]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:01.865161 18020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66eaf1357ed146ae97d0df37665752a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.865161 17880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66eaf1357ed146ae97d0df37665752a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.865176 18157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66eaf1357ed146ae97d0df37665752a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:01.867573 18250 tablet_bootstrap.cc:492] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: Bootstrap starting.
I20260619 11:05:01.867921 18249 tablet_bootstrap.cc:492] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: Bootstrap starting.
I20260619 11:05:01.868623 18250 tablet_bootstrap.cc:654] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.868717 18249 tablet_bootstrap.cc:654] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.869262 18251 tablet_bootstrap.cc:492] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: Bootstrap starting.
I20260619 11:05:01.870173 18251 tablet_bootstrap.cc:654] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:01.870236 18250 tablet_bootstrap.cc:492] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: No bootstrap required, opened a new log
I20260619 11:05:01.870303 18250 ts_tablet_manager.cc:1397] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:05:01.870782 18250 raft_consensus.cc:348] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.870900 18250 raft_consensus.cc:374] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.870942 18250 raft_consensus.cc:729] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a647dd379d354afa8a2e21c5a25b46e4, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.871073 18249 tablet_bootstrap.cc:492] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: No bootstrap required, opened a new log
I20260619 11:05:01.871059 18250 consensus_queue.cc:260] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.871155 18249 ts_tablet_manager.cc:1397] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:05:01.871311 18257 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.871451 18257 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:01.871316 18250 ts_tablet_manager.cc:1428] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:01.871573 18249 raft_consensus.cc:348] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.871744 18249 raft_consensus.cc:374] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.871783 18249 raft_consensus.cc:729] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9cebfc5fe0406e8fc10aa320fd812b, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.871907 18249 consensus_queue.cc:260] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.872195 18249 ts_tablet_manager.cc:1428] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:01.872139 18256 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.872339 18256 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:01.872836 18251 tablet_bootstrap.cc:492] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: No bootstrap required, opened a new log
I20260619 11:05:01.872925 18251 ts_tablet_manager.cc:1397] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 11:05:01.873420 18251 raft_consensus.cc:348] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.873548 18251 raft_consensus.cc:374] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:01.873597 18251 raft_consensus.cc:729] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4799d497f514e1d81d7b364b0d86876, State: Initialized, Role: FOLLOWER
I20260619 11:05:01.873729 18251 consensus_queue.cc:260] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.874020 18255 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.874188 18255 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:01.874032 18251 ts_tablet_manager.cc:1428] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:05:01.886047 18255 raft_consensus.cc:482] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:01.886147 18255 raft_consensus.cc:504] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.887171 18255 leader_election.cc:288] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9cebfc5fe0406e8fc10aa320fd812b (127.16.61.1:36051), a647dd379d354afa8a2e21c5a25b46e4 (127.16.61.3:37837)
I20260619 11:05:01.891348 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.887577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:37837 (local address 127.16.61.2:51791)
0619 11:05:01.887892 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.887906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.887992 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.888241 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.888244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.888259 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.888371 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.888378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.889187 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.889191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.890137 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.890145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.890916 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.890920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.891021 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.891046 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.891112 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.891190 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":24,"thread_start_us":61,"threads_started":1}
I20260619 11:05:01.891340 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.887490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:57321 (local address 127.16.61.1:36051)
0619 11:05:01.887605 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.887611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.887714 (+ 103us) server_negotiation.cc:407] Connection header received
0619 11:05:01.887878 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.887882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.887968 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.888068 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.888350 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.889135 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.890276 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.890759 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.890857 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.890922 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.891169 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.891241 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:05:01.891444 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.887704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:51791 (local address 127.16.61.3:37837)
0619 11:05:01.888018 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.888025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.888040 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:05:01.888087 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.888090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.888133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.888223 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:01.888486 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.889078 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.890251 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.890771 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.890906 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.891192 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.891234 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.891304 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":87,"threads_started":1}
I20260619 11:05:01.891744 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.887340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:36051 (local address 127.16.61.2:57321)
0619 11:05:01.887595 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.887609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.887749 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.888079 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.888081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.888098 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:01.888197 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.888202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.889249 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.889255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.890145 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.890154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.890968 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.890971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.891046 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.891063 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.891477 (+ 414us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.891553 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":25,"thread_start_us":81,"threads_started":1}
I20260619 11:05:01.891819 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66eaf1357ed146ae97d0df37665752a3" candidate_uuid: "b4799d497f514e1d81d7b364b0d86876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a647dd379d354afa8a2e21c5a25b46e4" is_pre_election: true
I20260619 11:05:01.891961 18180 raft_consensus.cc:2454] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4799d497f514e1d81d7b364b0d86876 in term 0.
I20260619 11:05:01.892167 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66eaf1357ed146ae97d0df37665752a3" candidate_uuid: "b4799d497f514e1d81d7b364b0d86876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" is_pre_election: true
I20260619 11:05:01.892298 17892 raft_consensus.cc:2454] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4799d497f514e1d81d7b364b0d86876 in term 0.
I20260619 11:05:01.892295 17971 leader_election.cc:302] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a647dd379d354afa8a2e21c5a25b46e4, b4799d497f514e1d81d7b364b0d86876; no voters:
I20260619 11:05:01.892570 18255 raft_consensus.cc:2781] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:01.892635 18255 raft_consensus.cc:482] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:01.892673 18255 raft_consensus.cc:3037] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.893610 18255 raft_consensus.cc:504] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.893937 18255 leader_election.cc:288] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [CANDIDATE]: Term 1 election: Requested vote from peers bf9cebfc5fe0406e8fc10aa320fd812b (127.16.61.1:36051), a647dd379d354afa8a2e21c5a25b46e4 (127.16.61.3:37837)
I20260619 11:05:01.894236 17892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66eaf1357ed146ae97d0df37665752a3" candidate_uuid: "b4799d497f514e1d81d7b364b0d86876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b"
I20260619 11:05:01.894379 17892 raft_consensus.cc:3037] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.894317 18180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66eaf1357ed146ae97d0df37665752a3" candidate_uuid: "b4799d497f514e1d81d7b364b0d86876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a647dd379d354afa8a2e21c5a25b46e4"
I20260619 11:05:01.894466 18180 raft_consensus.cc:3037] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:01.895287 17892 raft_consensus.cc:2454] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4799d497f514e1d81d7b364b0d86876 in term 1.
I20260619 11:05:01.895438 18180 raft_consensus.cc:2454] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4799d497f514e1d81d7b364b0d86876 in term 1.
I20260619 11:05:01.895596 17974 leader_election.cc:302] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4799d497f514e1d81d7b364b0d86876, bf9cebfc5fe0406e8fc10aa320fd812b; no voters:
I20260619 11:05:01.895800 18255 raft_consensus.cc:2781] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:01.896003 18255 raft_consensus.cc:686] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [term 1 LEADER]: Becoming Leader. State: Replica: b4799d497f514e1d81d7b364b0d86876, State: Running, Role: LEADER
I20260619 11:05:01.896263 18255 consensus_queue.cc:237] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } }
I20260619 11:05:01.897428 18296 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: TxnStatusTablet state changed. Reason: New leader b4799d497f514e1d81d7b364b0d86876. Latest consensus state: current_term: 1 leader_uuid: "b4799d497f514e1d81d7b364b0d86876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.897540 18296 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:01.897785 18298 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:05:01.897940 18298 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:05:01.898861 17743 catalog_manager.cc:5495] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4799d497f514e1d81d7b364b0d86876 (127.16.61.2). New cstate: current_term: 1 leader_uuid: "b4799d497f514e1d81d7b364b0d86876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:01.916779 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.913196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.16.61.2:46107)
0619 11:05:01.913342 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.913346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.913361 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:05:01.913520 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.913524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.913580 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.913678 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:01.913958 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.914464 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.915321 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.915539 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.915652 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.915979 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:01.916387 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:01.916515 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.916584 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.916633 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 11:05:01.916780 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.913070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:46107 (local address 127.0.0.1:43618)
0619 11:05:01.913206 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.913250 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.913360 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.913705 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.913707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.913717 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:01.913846 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.913851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.914575 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.914580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.915193 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.915200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.915650 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.915653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.915779 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.915834 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:01.916487 (+ 653us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:01.916503 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.916574 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.916655 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 11:05:01.919806 18180 raft_consensus.cc:1261] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Refusing update from remote peer b4799d497f514e1d81d7b364b0d86876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:01.920028 17892 raft_consensus.cc:1261] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Refusing update from remote peer b4799d497f514e1d81d7b364b0d86876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:01.920327 18255 consensus_queue.cc:1035] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:01.920496 18296 consensus_queue.cc:1035] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:01.922221 18257 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: TxnStatusTablet state changed. Reason: New leader b4799d497f514e1d81d7b364b0d86876. Latest consensus state: current_term: 1 leader_uuid: "b4799d497f514e1d81d7b364b0d86876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.922327 18257 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:01.924127 18256 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: TxnStatusTablet state changed. Reason: New leader b4799d497f514e1d81d7b364b0d86876. Latest consensus state: current_term: 1 leader_uuid: "b4799d497f514e1d81d7b364b0d86876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.924243 18256 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:01.925753 18296 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4799d497f514e1d81d7b364b0d86876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.925877 18296 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:01.925952 18257 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4799d497f514e1d81d7b364b0d86876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.925999 18296 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4799d497f514e1d81d7b364b0d86876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.926050 18257 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P a647dd379d354afa8a2e21c5a25b46e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:01.926070 18296 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P b4799d497f514e1d81d7b364b0d86876: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:01.926927 18256 tablet_replica.cc:439] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4799d497f514e1d81d7b364b0d86876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 } } peers { permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 } } peers { permanent_uuid: "a647dd379d354afa8a2e21c5a25b46e4" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37837 } } }
I20260619 11:05:01.927038 18256 tablet_replica.cc:442] T 66eaf1357ed146ae97d0df37665752a3 P bf9cebfc5fe0406e8fc10aa320fd812b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:01.937789 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.933164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.16.61.3:37837)
0619 11:05:01.933272 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.933278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.933426 (+ 148us) server_negotiation.cc:407] Connection header received
0619 11:05:01.933549 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.933552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.933634 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.933743 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:01.933960 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.935009 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.935956 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.936292 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.936396 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.936654 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:01.937226 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:01.937350 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.937577 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.937640 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 11:05:01.951323 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.945509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:38517 (local address 127.16.61.2:46107)
0619 11:05:01.945639 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.945643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.948254 (+ 2611us) server_negotiation.cc:407] Connection header received
0619 11:05:01.948421 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.948425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.948479 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.948567 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:01.948840 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.949340 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.950247 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.950478 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.950598 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.950892 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:01.950960 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:01.951042 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.951135 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.951181 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:05:01.951346 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.945353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:46107 (local address 127.16.61.3:38517)
0619 11:05:01.948133 (+ 2780us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.948146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.948297 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.948578 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.948580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.948589 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:01.948720 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.948726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.949452 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.949458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.950117 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.950125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.950599 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.950602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.950703 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.950743 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:01.951057 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:01.951066 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.951126 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.951202 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2690,"thread_start_us":85,"threads_started":1}
I20260619 11:05:01.958374 18040 raft_consensus.cc:1261] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Refusing update from remote peer a647dd379d354afa8a2e21c5a25b46e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:01.958374 17892 raft_consensus.cc:1261] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Refusing update from remote peer a647dd379d354afa8a2e21c5a25b46e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:01.958847 18257 consensus_queue.cc:1035] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4799d497f514e1d81d7b364b0d86876" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 46107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:01.959254 18257 consensus_queue.cc:1035] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9cebfc5fe0406e8fc10aa320fd812b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:01.984151 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.980049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:51853 (local address 127.16.61.3:37837)
0619 11:05:01.980172 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:01.980177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:01.980421 (+ 244us) server_negotiation.cc:407] Connection header received
0619 11:05:01.980616 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:01.980619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:01.980665 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:01.980768 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:01.981011 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.981642 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.982924 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.983234 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.983293 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.983694 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:01.983785 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:01.983878 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:01.983971 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 11:05:01.984016 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:05:01.984166 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:01.979723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:37837 (local address 127.16.61.2:51853)
0619 11:05:01.980342 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:01.980358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:01.980499 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:01.980778 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:01.980781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:01.980789 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:01.980894 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.980900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.981906 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.981910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.982801 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:01.982811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:01.983425 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:01.983427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:01.983483 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:01.983520 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:01.983880 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:01.983889 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 11:05:01.983953 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:05:01.984027 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":102,"threads_started":1}
I20260619 11:05:02.013587 17741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:02.013784 17741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:02.016222 17741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3b4b9b1baac4c7895c1001aae58cb54: Sending DeleteTablet for 3 replicas of tablet 2bbeed03b8294322862b35d4705f0c93
I20260619 11:05:02.016784 17880 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bbeed03b8294322862b35d4705f0c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:02 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:59256
I20260619 11:05:02.017053 18020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bbeed03b8294322862b35d4705f0c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:02 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:44494
I20260619 11:05:02.017352 18322 tablet_replica.cc:330] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: stopping tablet replica
I20260619 11:05:02.017462 18157 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bbeed03b8294322862b35d4705f0c93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:02 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:43270
I20260619 11:05:02.017524 18322 raft_consensus.cc:2229] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:02.017743 18322 raft_consensus.cc:2258] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:02.018456 18322 ts_tablet_manager.cc:1905] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:02.019039 18323 tablet_replica.cc:330] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: stopping tablet replica
I20260619 11:05:02.019171 18323 raft_consensus.cc:2229] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:02.019297 18323 raft_consensus.cc:2258] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:02.019544 18324 tablet_replica.cc:330] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: stopping tablet replica
I20260619 11:05:02.019745 18324 raft_consensus.cc:2229] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:02.020078 18324 raft_consensus.cc:2258] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:02.021124 18323 ts_tablet_manager.cc:1905] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:02.022145 18322 ts_tablet_manager.cc:1918] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:02.022230 18322 log.cc:1196] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/2bbeed03b8294322862b35d4705f0c93
I20260619 11:05:02.022490 18322 ts_tablet_manager.cc:1939] T 2bbeed03b8294322862b35d4705f0c93 P bf9cebfc5fe0406e8fc10aa320fd812b: Deleting consensus metadata
I20260619 11:05:02.023355 17730 catalog_manager.cc:4841] TS bf9cebfc5fe0406e8fc10aa320fd812b (127.16.61.1:36051): tablet 2bbeed03b8294322862b35d4705f0c93 (table test-table [id=79b23f0911294524a8bd9993872eb26e]) successfully deleted
I20260619 11:05:02.023948 18323 ts_tablet_manager.cc:1918] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:02.024037 18323 log.cc:1196] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/2bbeed03b8294322862b35d4705f0c93
I20260619 11:05:02.024070 18324 ts_tablet_manager.cc:1905] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:02.024320 18323 ts_tablet_manager.cc:1939] T 2bbeed03b8294322862b35d4705f0c93 P b4799d497f514e1d81d7b364b0d86876: Deleting consensus metadata
I20260619 11:05:02.025032 17727 catalog_manager.cc:4841] TS b4799d497f514e1d81d7b364b0d86876 (127.16.61.2:46107): tablet 2bbeed03b8294322862b35d4705f0c93 (table test-table [id=79b23f0911294524a8bd9993872eb26e]) successfully deleted
I20260619 11:05:02.027302 18324 ts_tablet_manager.cc:1918] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:02.027388 18324 log.cc:1196] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/2bbeed03b8294322862b35d4705f0c93
I20260619 11:05:02.027643 18324 ts_tablet_manager.cc:1939] T 2bbeed03b8294322862b35d4705f0c93 P a647dd379d354afa8a2e21c5a25b46e4: Deleting consensus metadata
I20260619 11:05:02.028365 17727 catalog_manager.cc:4841] TS a647dd379d354afa8a2e21c5a25b46e4 (127.16.61.3:37837): tablet 2bbeed03b8294322862b35d4705f0c93 (table test-table [id=79b23f0911294524a8bd9993872eb26e]) successfully deleted
Jun 19 11:05:02 dist-test-slave-8vcz krb5kdc[17698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867101, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:05:02.037580 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:02.020530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.16.61.1:36051)
0619 11:05:02.020654 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:02.020658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:02.025182 (+ 4524us) server_negotiation.cc:407] Connection header received
0619 11:05:02.025401 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:02.025404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:02.025457 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:02.025509 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:02.026483 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:02.027293 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:02.028515 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:02.028874 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:02.028949 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:02.032898 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:02.032910 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:02.032927 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:02.032958 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:02.035114 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:02.035651 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:02.035657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:02.035661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:02.035785 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:02.036104 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:02.036110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:02.036113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:02.036447 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:02.037046 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:02.037269 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 11:05:02.037403 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 11:05:02.038022 17940 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48604
I20260619 11:05:02.049454 18289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:02.039898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.16.61.62:40143)
0619 11:05:02.040022 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:02.040027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:02.040429 (+ 402us) server_negotiation.cc:407] Connection header received
0619 11:05:02.040547 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:02.040550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:02.040600 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:02.040684 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:02.041604 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:02.042483 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:02.043577 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:02.044009 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:02.044115 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:02.045749 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:02.045753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:02.045756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:02.045782 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:02.047601 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:02.048118 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:02.048122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:02.048124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:02.048210 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:02.048517 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:02.048521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:02.048522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:02.048725 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:02.048855 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:02.049157 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 11:05:02.049278 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 11:05:02.050206 17741 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44142
I20260619 11:05:02.051731 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 17815
I20260619 11:05:02.060859 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 17955
I20260619 11:05:02.068351 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 18095
I20260619 11:05:02.074246 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 17711
2026-06-19T11:05:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2941 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:02 dist-test-slave-8vcz krb5kdc[18331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:02 dist-test-slave-8vcz krb5kdc[18331](info): set up 2 sockets
Jun 19 11:05:02 dist-test-slave-8vcz krb5kdc[18331](info): commencing operation
krb5kdc: starting...
W20260619 11:05:03.139894 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:03 dist-test-slave-8vcz krb5kdc[18331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:03.262194 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35307
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:35307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:03.370159 18344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:03.370443 18344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:03.370488 18344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:03.374446 18344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:03.374498 18344 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:05:03.374516 18344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:03.374533 18344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:03.374549 18344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:03.374564 18344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:03.379606 18344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:35307
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:03.380530 18344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:03.381429 18344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:03.389673 18352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:03.389657 18350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:03.389724 18344 server_base.cc:1034] running on GCE node
W20260619 11:05:03.389657 18349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:03.390548 18344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:03.391602 18344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:03.392818 18344 hybrid_clock.cc:648] HybridClock initialized: now 1781867103392800 us; error 30 us; skew 500 ppm
Jun 19 11:05:03 dist-test-slave-8vcz krb5kdc[18331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867103, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:03.395503 18344 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:03.396384 18344 webserver.cc:460] Webserver started at http://127.16.61.62:45081/ using document root <none> and password file <none>
I20260619 11:05:03.396925 18344 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:03.397020 18344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:03.397251 18344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:03.399184 18344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "148eff810f04433882fe37bbfceea5a6"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "6384304b5de3a33e3a618e8ea096e1d5"
server_key_iv: "8eed69b52f783cf9e2819ae3a76992a3"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.399802 18344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "148eff810f04433882fe37bbfceea5a6"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "6384304b5de3a33e3a618e8ea096e1d5"
server_key_iv: "8eed69b52f783cf9e2819ae3a76992a3"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.403339 18344 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 11:05:03.405802 18359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:03.407012 18344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:05:03.407156 18344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "148eff810f04433882fe37bbfceea5a6"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "6384304b5de3a33e3a618e8ea096e1d5"
server_key_iv: "8eed69b52f783cf9e2819ae3a76992a3"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.407292 18344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:03.423298 18344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:03.426159 18344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:03.426363 18344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:03.433935 18344 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:35307
I20260619 11:05:03.433959 18421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:35307 every 8 connection(s)
I20260619 11:05:03.434877 18344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:03.437736 18422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:03.437958 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 18344
I20260619 11:05:03.438051 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:03.438328 16628 external_mini_cluster.cc:1428] Setting key 49ae1a6177c98914104ba4a48abccbff
I20260619 11:05:03.445446 18422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: Bootstrap starting.
I20260619 11:05:03.448274 18422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: Neither blocks nor log segments found. Creating new log.
Jun 19 11:05:03 dist-test-slave-8vcz krb5kdc[18331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:03.449214 18422 log.cc:824] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:03.451160 18422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: No bootstrap required, opened a new log
I20260619 11:05:03.454387 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.439783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.16.61.62:35307)
0619 11:05:03.440195 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:03.440215 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:03.440263 (+ 48us) server_negotiation.cc:407] Connection header received
0619 11:05:03.441399 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:03.441424 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:03.441882 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:03.442274 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:03.442895 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.444569 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.445399 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.446116 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.446282 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.449098 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:03.449103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:03.449120 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:03.449146 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:03.451493 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.452065 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.452070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.452074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.452199 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.452528 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.452530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.452531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.452901 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:03.453039 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:03.453375 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 11:05:03.453682 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
I20260619 11:05:03.454936 18422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148eff810f04433882fe37bbfceea5a6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35307 } }
I20260619 11:05:03.455179 18422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:03.455260 18422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148eff810f04433882fe37bbfceea5a6, State: Initialized, Role: FOLLOWER
I20260619 11:05:03.456728 18422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148eff810f04433882fe37bbfceea5a6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35307 } }
I20260619 11:05:03.456864 18422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:03.456956 18422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:03.457082 18422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:03.457994 18422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148eff810f04433882fe37bbfceea5a6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35307 } }
I20260619 11:05:03.458359 18422 leader_election.cc:302] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 148eff810f04433882fe37bbfceea5a6; no voters:
I20260619 11:05:03.458685 18422 leader_election.cc:288] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:03.458858 18427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:03.459107 18427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [term 1 LEADER]: Becoming Leader. State: Replica: 148eff810f04433882fe37bbfceea5a6, State: Running, Role: LEADER
I20260619 11:05:03.459383 18427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148eff810f04433882fe37bbfceea5a6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35307 } }
I20260619 11:05:03.460208 18422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:03.460605 18429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 148eff810f04433882fe37bbfceea5a6. Latest consensus state: current_term: 1 leader_uuid: "148eff810f04433882fe37bbfceea5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148eff810f04433882fe37bbfceea5a6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35307 } } }
I20260619 11:05:03.460748 18429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:03.461061 18428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "148eff810f04433882fe37bbfceea5a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "148eff810f04433882fe37bbfceea5a6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35307 } } }
I20260619 11:05:03.461159 18428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:03.461508 18431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:03.464422 18431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:03.469681 18431 catalog_manager.cc:1347] Generated new cluster ID: b4c1bcb51403468996159ea71ec253b7
I20260619 11:05:03.469758 18431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:03.476384 18431 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:03.477313 18431 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:03.481607 18431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: Generated new TSK 0
I20260619 11:05:03.482227 18431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:03.514184 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35307
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:03.625052 18448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:03.625306 18448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:03.625353 18448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:03.629372 18448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:03.629423 18448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:03.629463 18448 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:05:03.629477 18448 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:03.629508 18448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:03.634557 18448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:35307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:03.635502 18448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:03.636530 18448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:03.646292 18453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:03.646343 18454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:03.646399 18448 server_base.cc:1034] running on GCE node
W20260619 11:05:03.646687 18456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:03.647110 18448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:03.647789 18448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:03.648978 18448 hybrid_clock.cc:648] HybridClock initialized: now 1781867103648965 us; error 28 us; skew 500 ppm
Jun 19 11:05:03 dist-test-slave-8vcz krb5kdc[18331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867103, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:03.651585 18448 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:03.652455 18448 webserver.cc:460] Webserver started at http://127.16.61.1:45959/ using document root <none> and password file <none>
I20260619 11:05:03.652949 18448 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:03.653005 18448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:03.653157 18448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:03.655027 18448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "21b90faf51224e6aa6b1d377c066f71d"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "4bc7005b320505dd5beee9b5607c7a65"
server_key_iv: "659570639db7dfa55bb2c2d4c721dc40"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.655505 18448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "21b90faf51224e6aa6b1d377c066f71d"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "4bc7005b320505dd5beee9b5607c7a65"
server_key_iv: "659570639db7dfa55bb2c2d4c721dc40"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.658962 18448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:05:03.661389 18463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:03.662642 18448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:03.662730 18448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "21b90faf51224e6aa6b1d377c066f71d"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "4bc7005b320505dd5beee9b5607c7a65"
server_key_iv: "659570639db7dfa55bb2c2d4c721dc40"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.662811 18448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:03.696192 18448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:03.699028 18448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:03.699175 18448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:03.700640 18448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:03.700690 18448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:03.700762 18448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:03.700802 18448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:05:03 dist-test-slave-8vcz krb5kdc[18331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867103, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:03.711894 18448 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:39479
I20260619 11:05:03.712576 18578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:39479 every 8 connection(s)
I20260619 11:05:03.712906 18448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:03.714516 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.701636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:32845 (local address 127.16.61.62:35307)
0619 11:05:03.701758 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:03.701761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:03.702771 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 11:05:03.703951 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:03.703954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:03.704015 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:03.704113 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:03.705317 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.705833 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.706999 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.707309 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.707433 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.710505 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:03.710508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:03.710511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:03.710537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:03.712329 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.712879 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.712882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.712883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.712957 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.713351 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.713356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.713358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.713637 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:03.713735 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:03.714215 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 11:05:03.714334 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:05:03.715247 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.702147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35307 (local address 127.16.61.1:32845)
0619 11:05:03.702623 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:03.702664 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:03.703691 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:03.704276 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:03.704282 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:03.704729 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:03.705168 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:03.705180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.705969 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.705973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:03.706866 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:03.706876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.707475 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.707479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:03.707721 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.708379 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:03.708405 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:03.710317 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:03.712448 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:03.712453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:03.712466 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:03.712774 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:03.713096 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:03.713100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:03.713102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:03.713247 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:03.713733 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:03.713740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:03.713978 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 11:05:03.714163 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 11:05:03.714464 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":141,"threads_started":1}
I20260619 11:05:03.716570 18579 heartbeater.cc:347] Connected to a master server at 127.16.61.62:35307
I20260619 11:05:03.716729 18579 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:03.717262 18579 heartbeater.cc:511] Master 127.16.61.62:35307 requested a full tablet report, sending...
I20260619 11:05:03.718827 18375 ts_manager.cc:194] Registered new tserver with Master: 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479)
I20260619 11:05:03.720177 18375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:32845
I20260619 11:05:03.720479 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 18448
I20260619 11:05:03.720599 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:03.720885 16628 external_mini_cluster.cc:1428] Setting key 61ed2a71182f2ff771c4c39f4a56504f
I20260619 11:05:03.726142 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.718460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35307 (local address 127.16.61.1:41169)
0619 11:05:03.718740 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:03.718755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:03.718851 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:03.719129 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:03.719131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:03.719320 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:03.719445 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:03.719452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.720432 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.720435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:03.721326 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:03.721335 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.722011 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.722014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:03.722101 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.722563 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:03.722579 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:03.722904 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:03.724707 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:03.724712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:03.724715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:03.724970 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:03.725230 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:03.725234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:03.725236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:03.725331 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:03.725758 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:03.725760 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:03.725829 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 11:05:03.725910 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:05:03.726009 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260619 11:05:03.726351 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.718578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:41169 (local address 127.16.61.62:35307)
0619 11:05:03.718678 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:03.718681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:03.718830 (+ 149us) server_negotiation.cc:407] Connection header received
0619 11:05:03.718959 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:03.718965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:03.719010 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:03.719097 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:03.719584 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.720205 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.721575 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.721884 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.721982 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.723043 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:03.723050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:03.723055 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:03.723086 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:03.724596 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.725072 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.725074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.725075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.725142 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.725430 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.725434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.725437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.725664 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:03.725767 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:03.725991 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 11:05:03.726119 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":31}
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:03.748637 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35307
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:03.855456 18588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:03.855695 18588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:03.855741 18588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:03.860255 18588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:03.860306 18588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:03.860351 18588 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:05:03.860365 18588 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:03.860401 18588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:03.865641 18588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:35307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:03.866621 18588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:03.867556 18588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:03.876711 18593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:03.876761 18594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:03.876907 18588 server_base.cc:1034] running on GCE node
W20260619 11:05:03.876725 18596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:03.877441 18588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:03.878044 18588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:03.879204 18588 hybrid_clock.cc:648] HybridClock initialized: now 1781867103879191 us; error 27 us; skew 500 ppm
Jun 19 11:05:03 dist-test-slave-8vcz krb5kdc[18331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867103, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:03.881939 18588 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:03.882774 18588 webserver.cc:460] Webserver started at http://127.16.61.2:35359/ using document root <none> and password file <none>
I20260619 11:05:03.883265 18588 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:03.883322 18588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:03.883483 18588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:03.885526 18588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "8edd097397094c70bb45fa9f95b9fcee"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "72a82e4c660f673eb77e06e9c1c713ec"
server_key_iv: "b880591ce8e7a051e6f4791590f8275e"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.886008 18588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "8edd097397094c70bb45fa9f95b9fcee"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "72a82e4c660f673eb77e06e9c1c713ec"
server_key_iv: "b880591ce8e7a051e6f4791590f8275e"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.889570 18588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 11:05:03.891820 18603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:03.892982 18588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:03.893134 18588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "8edd097397094c70bb45fa9f95b9fcee"
format_stamp: "Formatted at 2026-06-19 11:05:03 on dist-test-slave-8vcz"
server_key: "72a82e4c660f673eb77e06e9c1c713ec"
server_key_iv: "b880591ce8e7a051e6f4791590f8275e"
server_key_version: "encryptionkey@0"
I20260619 11:05:03.893252 18588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:03.904564 18588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:03.907632 18588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:03.907866 18588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:03.909238 18588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:03.909314 18588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:03.909384 18588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:03.909443 18588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:05:03 dist-test-slave-8vcz krb5kdc[18331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867103, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:03.919938 18588 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:34473
I20260619 11:05:03.920006 18718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:34473 every 8 connection(s)
I20260619 11:05:03.920769 18588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:03.922967 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.909865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:53415 (local address 127.16.61.62:35307)
0619 11:05:03.910019 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:03.910025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:03.911406 (+ 1381us) server_negotiation.cc:407] Connection header received
0619 11:05:03.912826 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:03.912829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:03.912876 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:03.912941 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:03.914528 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.915180 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.916475 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.916794 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.916895 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.919504 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:03.919508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:03.919510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:03.919540 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:03.921163 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.921726 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.921728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.921730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.921804 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.922104 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.922106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.922107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.922269 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:03.922361 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:03.922765 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 11:05:03.922856 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:05:03.923774 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.910722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35307 (local address 127.16.61.2:53415)
0619 11:05:03.911230 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:03.911290 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:03.912599 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:03.913127 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:03.913138 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:03.913635 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:03.914362 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:03.914384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.915307 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.915310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:03.916352 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:03.916362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.916949 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.916953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:03.917109 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.917642 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:03.917657 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:03.919320 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:03.921317 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:03.921322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:03.921336 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:03.921636 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:03.921895 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:03.921897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:03.921898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:03.922019 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:03.922354 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:03.922359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:03.922558 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 11:05:03.922722 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 11:05:03.922960 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":141,"threads_started":1}
I20260619 11:05:03.924427 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 18588
I20260619 11:05:03.924544 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:03.924805 16628 external_mini_cluster.cc:1428] Setting key 588204664c254d149d542cc3ebed39c6
I20260619 11:05:03.925074 18719 heartbeater.cc:347] Connected to a master server at 127.16.61.62:35307
I20260619 11:05:03.925215 18719 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:03.925750 18719 heartbeater.cc:511] Master 127.16.61.62:35307 requested a full tablet report, sending...
I20260619 11:05:03.926733 18376 ts_manager.cc:194] Registered new tserver with Master: 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473)
I20260619 11:05:03.927505 18376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:53415
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
I20260619 11:05:03.934608 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.927132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:57671 (local address 127.16.61.62:35307)
0619 11:05:03.927253 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:03.927257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:03.927724 (+ 467us) server_negotiation.cc:407] Connection header received
0619 11:05:03.927927 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:03.927929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:03.927976 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:03.928067 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:03.928639 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.929269 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.930103 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.930319 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.930438 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.931531 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:03.931534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:03.931536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:03.931565 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:03.933012 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.933439 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.933443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.933445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.933523 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:03.933805 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:03.933810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:03.933812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:03.934004 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:03.934124 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:03.934372 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 11:05:03.934473 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:05:03.934624 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:03.927327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35307 (local address 127.16.61.2:57671)
0619 11:05:03.927621 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:03.927639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:03.927800 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:03.928082 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:03.928085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:03.928324 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:03.928516 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:03.928523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.929412 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.929416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:03.929982 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:03.929989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:03.930439 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:03.930443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:03.930539 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:03.931107 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:03.931118 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:03.931399 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:03.933147 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:03.933150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:03.933152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:03.933341 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:03.933627 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:03.933630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:03.933631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:03.933714 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:03.934114 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:03.934117 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:03.934201 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 11:05:03.934315 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 11:05:03.934446 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":5,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:05:03.954566 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35307
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:04.064308 18728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:04.064602 18728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:04.064678 18728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:04.068856 18728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:04.068938 18728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:04.069006 18728 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 11:05:04.069052 18728 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:04.069123 18728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:04.074584 18728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:35307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:04.075623 18728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:04.076625 18728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:04.086161 18734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:04.086174 18733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:04.086249 18728 server_base.cc:1034] running on GCE node
W20260619 11:05:04.086174 18736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:04.086776 18728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:04.087412 18728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:04.088567 18728 hybrid_clock.cc:648] HybridClock initialized: now 1781867104088544 us; error 36 us; skew 500 ppm
Jun 19 11:05:04 dist-test-slave-8vcz krb5kdc[18331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867104, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:04.091252 18728 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:04.092187 18728 webserver.cc:460] Webserver started at http://127.16.61.3:46345/ using document root <none> and password file <none>
I20260619 11:05:04.092772 18728 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:04.092831 18728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:04.093039 18728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:04.095008 18728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "491d97948e6c4dcd9f5247f1a6a8073c"
format_stamp: "Formatted at 2026-06-19 11:05:04 on dist-test-slave-8vcz"
server_key: "496bf74bcbf0dc3b889e1d2ea25c1a27"
server_key_iv: "b9e797c6c3ca2bb274dbff389019838a"
server_key_version: "encryptionkey@0"
I20260619 11:05:04.095556 18728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "491d97948e6c4dcd9f5247f1a6a8073c"
format_stamp: "Formatted at 2026-06-19 11:05:04 on dist-test-slave-8vcz"
server_key: "496bf74bcbf0dc3b889e1d2ea25c1a27"
server_key_iv: "b9e797c6c3ca2bb274dbff389019838a"
server_key_version: "encryptionkey@0"
I20260619 11:05:04.099534 18728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 11:05:04.102264 18743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:04.103853 18728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:04.103955 18728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "491d97948e6c4dcd9f5247f1a6a8073c"
format_stamp: "Formatted at 2026-06-19 11:05:04 on dist-test-slave-8vcz"
server_key: "496bf74bcbf0dc3b889e1d2ea25c1a27"
server_key_iv: "b9e797c6c3ca2bb274dbff389019838a"
server_key_version: "encryptionkey@0"
I20260619 11:05:04.104099 18728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:04.129418 18728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:04.133142 18728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:04.133374 18728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:04.135154 18728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:04.135219 18728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:04.135272 18728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:04.135293 18728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:05:04 dist-test-slave-8vcz krb5kdc[18331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867104, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:04.146202 18728 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:42387
I20260619 11:05:04.146262 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:42387 every 8 connection(s)
I20260619 11:05:04.147385 18728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:04.149433 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.136004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:42493 (local address 127.16.61.62:35307)
0619 11:05:04.136173 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.136181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.137048 (+ 867us) server_negotiation.cc:407] Connection header received
0619 11:05:04.138091 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.138097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.138157 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.138223 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:04.140024 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.140664 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.141875 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.142109 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.142231 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.145341 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:04.145345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:04.145348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:04.145376 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:04.147217 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:04.147949 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:04.147952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:04.147954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:04.148044 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:04.148441 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:04.148445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:04.148446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:04.148636 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:04.148714 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.149184 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.149319 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 11:05:04.150241 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.136421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35307 (local address 127.16.61.3:42493)
0619 11:05:04.136881 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.136931 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.137836 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.138743 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.138753 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.139192 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:04.139863 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.139887 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.140799 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.140804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.141738 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.141747 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.142282 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.142286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.142487 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.143169 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:04.143188 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:04.145151 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:04.147354 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:04.147359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:04.147370 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:04.147841 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:04.148163 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:04.148166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:04.148168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:04.148323 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:04.148732 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:04.148737 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:04.148958 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.149100 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.149380 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":134,"threads_started":1}
I20260619 11:05:04.151211 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 18728
I20260619 11:05:04.151348 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:04.151525 18859 heartbeater.cc:347] Connected to a master server at 127.16.61.62:35307
I20260619 11:05:04.151644 16628 external_mini_cluster.cc:1428] Setting key 6341dd61e1daf611a2b437048876300d
I20260619 11:05:04.151789 18859 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:04.152401 18859 heartbeater.cc:511] Master 127.16.61.62:35307 requested a full tablet report, sending...
I20260619 11:05:04.153298 18376 ts_manager.cc:194] Registered new tserver with Master: 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387)
I20260619 11:05:04.153981 18376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:42493
I20260619 11:05:04.160954 18865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.154077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35307 (local address 127.16.61.3:36915)
0619 11:05:04.154304 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.154318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.154405 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.154645 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.154648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.154853 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:04.154975 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.154982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.155779 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.155782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.156425 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.156434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.156857 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.156861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.156938 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.157382 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:04.157393 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:04.157695 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:04.159244 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:04.159247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:04.159248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:04.159437 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:04.159865 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:04.159868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:04.159870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:04.159954 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:04.160401 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:04.160403 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:04.160478 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.160571 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.160734 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":5,"thread_start_us":85,"threads_started":1}
I20260619 11:05:04.161054 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.154123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:36915 (local address 127.16.61.62:35307)
0619 11:05:04.154235 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.154242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.154399 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:05:04.154478 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.154482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.154525 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.154603 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:04.155109 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.155614 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.156532 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.156733 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.156811 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.157840 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:04.157843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:04.157845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:04.157865 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:04.159140 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:04.159547 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:04.159550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:04.159551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:04.159628 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:04.160091 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:04.160094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:04.160095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:04.160258 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:04.160340 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.160683 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.160858 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:05:04.165627 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:05:04 dist-test-slave-8vcz krb5kdc[18331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:05:04 dist-test-slave-8vcz krb5kdc[18331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:04.191326 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.183018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38380 (local address 127.16.61.62:35307)
0619 11:05:04.183146 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.183149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.183161 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:04.183199 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.183201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.183238 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.183301 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:04.184049 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.184574 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.185367 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.185587 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.185665 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.188166 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:04.188170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:04.188172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:04.188193 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:04.189554 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:04.190009 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:04.190012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:04.190013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:04.190097 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:04.190444 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:04.190447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:04.190449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:04.190618 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:04.190720 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.190989 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.191133 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:04.194321 18376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:04.197571 18376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:04.211994 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.206365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.16.61.1:39479)
0619 11:05:04.206717 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.206731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.206758 (+ 27us) server_negotiation.cc:407] Connection header received
0619 11:05:04.206855 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.206861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.207245 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.207447 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:04.207760 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.208999 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.210596 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.211152 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.211279 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.211443 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.211773 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.211872 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":79,"threads_started":1}
I20260619 11:05:04.212016 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.206465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.16.61.2:34473)
0619 11:05:04.206846 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.206855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.207155 (+ 300us) server_negotiation.cc:407] Connection header received
0619 11:05:04.207253 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.207260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.207446 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.207579 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:04.207908 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.208753 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.210272 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.211075 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.211177 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.211336 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.211764 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.211869 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":89,"threads_started":1}
I20260619 11:05:04.212016 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.205664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.16.61.3:42387)
0619 11:05:04.207384 (+ 1720us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.207396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.207417 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:05:04.207480 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.207485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.207731 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.207889 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:04.208114 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.209220 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.210593 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.211449 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.211595 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.211775 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.211817 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.211886 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1585,"thread_start_us":101,"threads_started":1}
I20260619 11:05:04.212664 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.205555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42387 (local address 127.0.0.1:35952)
0619 11:05:04.205972 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.206016 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.206131 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.207875 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.207878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.207896 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:04.207994 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.208000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.209329 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.209332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.210484 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.210494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.211568 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.211574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.211698 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.211722 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.211786 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.212544 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":727,"thread_start_us":83,"threads_started":1}
I20260619 11:05:04.212677 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.206088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34473 (local address 127.0.0.1:46946)
0619 11:05:04.207028 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.207042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.207123 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.207632 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.207635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.207654 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:04.207782 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.207788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.208873 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.208876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.210116 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.210125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.211400 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.211405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.211561 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.211583 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.211648 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.212482 (+ 834us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":784,"thread_start_us":88,"threads_started":1}
I20260619 11:05:04.212666 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.205609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:39479 (local address 127.0.0.1:55452)
0619 11:05:04.206457 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.206470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.206564 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.207446 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.207452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.207476 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:04.207599 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.207605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.209124 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.209127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.210453 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.210464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.211279 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.211283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.211383 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.211474 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.211608 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.212482 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":93,"thread_start_us":103,"threads_started":1}
I20260619 11:05:04.215372 18653 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae54cbbf6a74ef1b638a5029508477a (DEFAULT_TABLE table=test-table [id=4f9af1c8b7f44bb4a5e2ffdf5da3166b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:04.215369 18513 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae54cbbf6a74ef1b638a5029508477a (DEFAULT_TABLE table=test-table [id=4f9af1c8b7f44bb4a5e2ffdf5da3166b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:04.216456 18653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae54cbbf6a74ef1b638a5029508477a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.216463 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae54cbbf6a74ef1b638a5029508477a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.217729 18793 tablet_service.cc:1467] Processing CreateTablet for tablet 2ae54cbbf6a74ef1b638a5029508477a (DEFAULT_TABLE table=test-table [id=4f9af1c8b7f44bb4a5e2ffdf5da3166b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:04.218680 18793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ae54cbbf6a74ef1b638a5029508477a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.222644 18882 tablet_bootstrap.cc:492] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Bootstrap starting.
I20260619 11:05:04.224071 18883 tablet_bootstrap.cc:492] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Bootstrap starting.
I20260619 11:05:04.225524 18882 tablet_bootstrap.cc:654] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.226208 18883 tablet_bootstrap.cc:654] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.226492 18884 tablet_bootstrap.cc:492] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Bootstrap starting.
I20260619 11:05:04.227106 18882 log.cc:824] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:04.227151 18883 log.cc:824] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:04.228420 18884 tablet_bootstrap.cc:654] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.229041 18882 tablet_bootstrap.cc:492] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: No bootstrap required, opened a new log
I20260619 11:05:04.229126 18884 log.cc:824] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:04.229244 18882 ts_tablet_manager.cc:1397] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 11:05:04.229739 18883 tablet_bootstrap.cc:492] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: No bootstrap required, opened a new log
I20260619 11:05:04.229911 18883 ts_tablet_manager.cc:1397] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:05:04.230693 18884 tablet_bootstrap.cc:492] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: No bootstrap required, opened a new log
I20260619 11:05:04.230880 18884 ts_tablet_manager.cc:1397] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:05:04.233042 18882 raft_consensus.cc:348] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
I20260619 11:05:04.233309 18882 raft_consensus.cc:374] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.233359 18882 raft_consensus.cc:729] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b90faf51224e6aa6b1d377c066f71d, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.233656 18884 raft_consensus.cc:348] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
I20260619 11:05:04.233896 18884 raft_consensus.cc:374] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.233808 18882 consensus_queue.cc:260] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
I20260619 11:05:04.233953 18884 raft_consensus.cc:729] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491d97948e6c4dcd9f5247f1a6a8073c, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.233631 18883 raft_consensus.cc:348] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
I20260619 11:05:04.234215 18883 raft_consensus.cc:374] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.234278 18883 raft_consensus.cc:729] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edd097397094c70bb45fa9f95b9fcee, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.234364 18884 consensus_queue.cc:260] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
I20260619 11:05:04.234678 18579 heartbeater.cc:503] Master 127.16.61.62:35307 was elected leader, sending a full tablet report...
I20260619 11:05:04.234826 18882 ts_tablet_manager.cc:1428] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:05:04.235075 18859 heartbeater.cc:503] Master 127.16.61.62:35307 was elected leader, sending a full tablet report...
I20260619 11:05:04.235245 18884 ts_tablet_manager.cc:1428] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 11:05:04.235791 18883 consensus_queue.cc:260] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
I20260619 11:05:04.236428 18719 heartbeater.cc:503] Master 127.16.61.62:35307 was elected leader, sending a full tablet report...
I20260619 11:05:04.236716 18883 ts_tablet_manager.cc:1428] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260619 11:05:04.398491 18860 tablet.cc:2369] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:04.410179 18890 raft_consensus.cc:482] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:04.410437 18890 raft_consensus.cc:504] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
I20260619 11:05:04.411751 18890 leader_election.cc:288] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387), 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479)
I20260619 11:05:04.415791 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.412005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42387 (local address 127.16.61.2:47005)
0619 11:05:04.412200 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.412215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.412317 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.412630 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.412632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.412675 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:04.412801 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.412807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.413675 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.413679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.414544 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.414553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.415376 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.415379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.415452 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.415471 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.415533 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.415624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":24}
I20260619 11:05:04.415803 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.412188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:47005 (local address 127.16.61.3:42387)
0619 11:05:04.412291 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.412297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.412316 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:05:04.412461 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.412464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.412514 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.412617 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:04.412942 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.413567 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.414673 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.415259 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.415362 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.415430 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.415577 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.415651 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:05:04.416465 18813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae54cbbf6a74ef1b638a5029508477a" candidate_uuid: "8edd097397094c70bb45fa9f95b9fcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" is_pre_election: true
I20260619 11:05:04.416787 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.412005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:39479 (local address 127.16.61.2:47567)
0619 11:05:04.412576 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.412592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.412689 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.412963 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.412966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.412983 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:04.413091 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.413096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.413859 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.413861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.415494 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.415502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.416384 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.416387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.416489 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.416514 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.416583 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.416662 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":250,"thread_start_us":93,"threads_started":1}
I20260619 11:05:04.416787 18813 raft_consensus.cc:2454] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8edd097397094c70bb45fa9f95b9fcee in term 0.
I20260619 11:05:04.416867 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.412079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:47567 (local address 127.16.61.1:39479)
0619 11:05:04.412229 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.412235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.412686 (+ 451us) server_negotiation.cc:407] Connection header received
0619 11:05:04.412799 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.412803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.412863 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.412937 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:04.413197 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.413740 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.415650 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.416248 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.416353 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.416408 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.416663 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.416731 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 11:05:04.417307 18606 leader_election.cc:302] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491d97948e6c4dcd9f5247f1a6a8073c, 8edd097397094c70bb45fa9f95b9fcee; no voters:
I20260619 11:05:04.417634 18890 raft_consensus.cc:2781] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:04.417735 18890 raft_consensus.cc:482] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:04.417675 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae54cbbf6a74ef1b638a5029508477a" candidate_uuid: "8edd097397094c70bb45fa9f95b9fcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b90faf51224e6aa6b1d377c066f71d" is_pre_election: true
I20260619 11:05:04.417781 18890 raft_consensus.cc:3037] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.417941 18533 raft_consensus.cc:2454] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8edd097397094c70bb45fa9f95b9fcee in term 0.
I20260619 11:05:04.418880 18890 raft_consensus.cc:504] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
I20260619 11:05:04.419160 18890 leader_election.cc:288] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [CANDIDATE]: Term 1 election: Requested vote from peers 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387), 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479)
I20260619 11:05:04.419552 18813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae54cbbf6a74ef1b638a5029508477a" candidate_uuid: "8edd097397094c70bb45fa9f95b9fcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491d97948e6c4dcd9f5247f1a6a8073c"
I20260619 11:05:04.419543 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ae54cbbf6a74ef1b638a5029508477a" candidate_uuid: "8edd097397094c70bb45fa9f95b9fcee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b90faf51224e6aa6b1d377c066f71d"
I20260619 11:05:04.419654 18813 raft_consensus.cc:3037] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.419725 18533 raft_consensus.cc:3037] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.420684 18533 raft_consensus.cc:2454] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8edd097397094c70bb45fa9f95b9fcee in term 1.
I20260619 11:05:04.420979 18604 leader_election.cc:302] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b90faf51224e6aa6b1d377c066f71d, 8edd097397094c70bb45fa9f95b9fcee; no voters:
I20260619 11:05:04.421159 18890 raft_consensus.cc:2781] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:04.421253 18813 raft_consensus.cc:2454] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8edd097397094c70bb45fa9f95b9fcee in term 1.
I20260619 11:05:04.421381 18890 raft_consensus.cc:686] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 1 LEADER]: Becoming Leader. State: Replica: 8edd097397094c70bb45fa9f95b9fcee, State: Running, Role: LEADER
I20260619 11:05:04.421705 18890 consensus_queue.cc:237] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } }
W20260619 11:05:04.421983 18720 tablet.cc:2369] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:04.424858 18375 catalog_manager.cc:5495] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee reported cstate change: term changed from 0 to 1, leader changed from <none> to 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2). New cstate: current_term: 1 leader_uuid: "8edd097397094c70bb45fa9f95b9fcee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } health_report { overall_health: HEALTHY } } }
W20260619 11:05:04.464685 18580 tablet.cc:2369] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 11:05:04 dist-test-slave-8vcz krb5kdc[18331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867103, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:04.474036 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.465328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35307 (local address 127.0.0.1:38390)
0619 11:05:04.465596 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.465609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.465710 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.466011 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.466014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.466425 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:04.466515 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.466520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.467381 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.467383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.467961 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.467970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.468402 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.468405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.468471 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.469119 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:04.469130 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:04.470555 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:04.472401 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:04.472409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:04.472437 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:04.472668 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:04.472940 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:04.472942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:04.472943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:04.473019 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:04.473444 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:04.473449 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:04.473547 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.473640 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.473832 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260619 11:05:04.474040 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.465418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.16.61.62:35307)
0619 11:05:04.465573 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.465579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.465681 (+ 102us) server_negotiation.cc:407] Connection header received
0619 11:05:04.465798 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.465801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.465839 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.465948 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:04.466670 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.467274 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.468064 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.468264 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.468340 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.470705 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:04.470708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:04.470710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:04.470730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:04.472190 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:04.472768 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:04.472770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:04.472772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:04.472844 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:04.473102 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:04.473105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:04.473106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:04.473318 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:04.473406 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.473708 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.473913 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":81}
I20260619 11:05:04.477870 18375 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:38390:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:05:04.478629 18375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:04.483696 18793 tablet_service.cc:1467] Processing CreateTablet for tablet 34e358ac70b4426b840bf5f7c751c56d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82aab2bccb7d47cba13dffdf861d028d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:04.483595 18513 tablet_service.cc:1467] Processing CreateTablet for tablet 34e358ac70b4426b840bf5f7c751c56d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82aab2bccb7d47cba13dffdf861d028d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:04.483593 18653 tablet_service.cc:1467] Processing CreateTablet for tablet 34e358ac70b4426b840bf5f7c751c56d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82aab2bccb7d47cba13dffdf861d028d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:04.484350 18793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e358ac70b4426b840bf5f7c751c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.484350 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e358ac70b4426b840bf5f7c751c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.484354 18653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e358ac70b4426b840bf5f7c751c56d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.486956 18883 tablet_bootstrap.cc:492] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: Bootstrap starting.
I20260619 11:05:04.487068 18882 tablet_bootstrap.cc:492] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: Bootstrap starting.
I20260619 11:05:04.487068 18884 tablet_bootstrap.cc:492] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: Bootstrap starting.
I20260619 11:05:04.487879 18883 tablet_bootstrap.cc:654] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.488045 18884 tablet_bootstrap.cc:654] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.488045 18882 tablet_bootstrap.cc:654] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.489012 18883 tablet_bootstrap.cc:492] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: No bootstrap required, opened a new log
I20260619 11:05:04.489104 18883 ts_tablet_manager.cc:1397] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:04.489365 18884 tablet_bootstrap.cc:492] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: No bootstrap required, opened a new log
I20260619 11:05:04.489459 18884 ts_tablet_manager.cc:1397] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:04.489573 18883 raft_consensus.cc:348] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.489672 18883 raft_consensus.cc:374] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.489702 18883 raft_consensus.cc:729] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edd097397094c70bb45fa9f95b9fcee, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.489704 18882 tablet_bootstrap.cc:492] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: No bootstrap required, opened a new log
I20260619 11:05:04.489777 18882 ts_tablet_manager.cc:1397] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 11:05:04.489814 18883 consensus_queue.cc:260] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.489881 18884 raft_consensus.cc:348] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.489969 18884 raft_consensus.cc:374] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.489998 18884 raft_consensus.cc:729] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491d97948e6c4dcd9f5247f1a6a8073c, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.490094 18883 ts_tablet_manager.cc:1428] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:04.490111 18884 consensus_queue.cc:260] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.490231 18882 raft_consensus.cc:348] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.490293 18890 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.490360 18882 raft_consensus.cc:374] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.490355 18884 ts_tablet_manager.cc:1428] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:04.490396 18882 raft_consensus.cc:729] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b90faf51224e6aa6b1d377c066f71d, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.490415 18890 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:04.490514 18882 consensus_queue.cc:260] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.490585 18889 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.490715 18889 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:04.490765 18882 ts_tablet_manager.cc:1428] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:05:04.490808 18888 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.490916 18888 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:04.646315 18889 raft_consensus.cc:482] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:04.646514 18889 raft_consensus.cc:504] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.647652 18889 leader_election.cc:288] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479), 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473)
I20260619 11:05:04.652232 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.648173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:34825 (local address 127.16.61.1:39479)
0619 11:05:04.648315 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.648321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.648499 (+ 178us) server_negotiation.cc:407] Connection header received
0619 11:05:04.648578 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.648581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.648626 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.648712 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:04.649002 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.649596 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.650937 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.651501 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.651611 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.651704 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.652032 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.652103 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:05:04.652232 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.648048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:51359 (local address 127.16.61.2:34473)
0619 11:05:04.648188 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.648194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.648212 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:05:04.648341 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.648345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.648402 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.648491 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:04.648789 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.649389 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.650868 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.651501 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.651611 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.651701 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.652030 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.652103 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:05:04.652275 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.647953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34473 (local address 127.16.61.3:51359)
0619 11:05:04.648102 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.648120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.648226 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.648521 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.648526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.648547 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:04.648668 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.648679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.649511 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.649516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.650744 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.650754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.651641 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.651646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.651872 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.651921 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.652000 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.652118 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":100}
I20260619 11:05:04.652290 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.647953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:39479 (local address 127.16.61.3:34825)
0619 11:05:04.648380 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.648393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.648499 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.648754 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.648758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.648775 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:04.648891 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.648896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.649705 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.649708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.650822 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.650833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.651650 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.651653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.651876 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.651927 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.652000 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.652118 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":173,"thread_start_us":96,"threads_started":1}
I20260619 11:05:04.652882 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e358ac70b4426b840bf5f7c751c56d" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b90faf51224e6aa6b1d377c066f71d" is_pre_election: true
I20260619 11:05:04.653046 18533 raft_consensus.cc:2454] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 0.
I20260619 11:05:04.653359 18673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e358ac70b4426b840bf5f7c751c56d" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd097397094c70bb45fa9f95b9fcee" is_pre_election: true
I20260619 11:05:04.653378 18744 leader_election.cc:302] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b90faf51224e6aa6b1d377c066f71d, 491d97948e6c4dcd9f5247f1a6a8073c; no voters:
I20260619 11:05:04.653594 18673 raft_consensus.cc:2454] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 0.
I20260619 11:05:04.653666 18889 raft_consensus.cc:2781] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:04.653735 18889 raft_consensus.cc:482] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:04.653787 18889 raft_consensus.cc:3037] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.654794 18889 raft_consensus.cc:504] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.655192 18889 leader_election.cc:288] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 election: Requested vote from peers 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479), 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473)
I20260619 11:05:04.655545 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e358ac70b4426b840bf5f7c751c56d" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b90faf51224e6aa6b1d377c066f71d"
I20260619 11:05:04.655691 18533 raft_consensus.cc:3037] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.655773 18673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e358ac70b4426b840bf5f7c751c56d" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd097397094c70bb45fa9f95b9fcee"
I20260619 11:05:04.655881 18673 raft_consensus.cc:3037] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.656629 18533 raft_consensus.cc:2454] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 1.
I20260619 11:05:04.656877 18673 raft_consensus.cc:2454] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 1.
I20260619 11:05:04.656899 18744 leader_election.cc:302] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b90faf51224e6aa6b1d377c066f71d, 491d97948e6c4dcd9f5247f1a6a8073c; no voters:
I20260619 11:05:04.657080 18889 raft_consensus.cc:2781] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:04.657308 18889 raft_consensus.cc:686] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 LEADER]: Becoming Leader. State: Replica: 491d97948e6c4dcd9f5247f1a6a8073c, State: Running, Role: LEADER
I20260619 11:05:04.657706 18889 consensus_queue.cc:237] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.660012 18904 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: TxnStatusTablet state changed. Reason: New leader 491d97948e6c4dcd9f5247f1a6a8073c. Latest consensus state: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.660132 18904 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:04.660404 18906 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:05:04.660604 18906 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:05:04.661222 18376 catalog_manager.cc:5495] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c reported cstate change: term changed from 0 to 1, leader changed from <none> to 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3). New cstate: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } health_report { overall_health: HEALTHY } } }
I20260619 11:05:04.680802 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.676761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.16.61.3:42387)
0619 11:05:04.676868 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.676872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.676886 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:05:04.677058 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.677061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.677120 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.677224 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:04.677433 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.678203 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.679091 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.679423 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.679523 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.679874 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.680363 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.680476 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.680567 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.680620 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:05:04.680821 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.676640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42387 (local address 127.0.0.1:35956)
0619 11:05:04.676814 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.676831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.676921 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.677242 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.677244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.677249 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:04.677334 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.677338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.678336 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.678339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.678957 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.678965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.679548 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.679550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.679623 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.679729 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.680481 (+ 752us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.680495 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.680556 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.680626 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 11:05:04.685988 18673 raft_consensus.cc:1261] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Refusing update from remote peer 491d97948e6c4dcd9f5247f1a6a8073c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:04.686153 18533 raft_consensus.cc:1261] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Refusing update from remote peer 491d97948e6c4dcd9f5247f1a6a8073c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:04.686671 18904 consensus_queue.cc:1035] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:04.686911 18889 consensus_queue.cc:1035] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:04.690191 18890 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: TxnStatusTablet state changed. Reason: New leader 491d97948e6c4dcd9f5247f1a6a8073c. Latest consensus state: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.690308 18890 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:04.691962 18888 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: TxnStatusTablet state changed. Reason: New leader 491d97948e6c4dcd9f5247f1a6a8073c. Latest consensus state: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.692112 18888 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:04.693293 18889 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.693401 18889 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:04.693694 18910 mvcc.cc:205] Tried to move back new op lower bound from 7298527660786925568 to 7298527660680790016. Current Snapshot: MvccSnapshot[applied={T|T < 7298527660786925568}]
I20260619 11:05:04.693958 18890 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.693996 18909 mvcc.cc:205] Tried to move back new op lower bound from 7298527660786925568 to 7298527660680790016. Current Snapshot: MvccSnapshot[applied={T|T < 7298527660786925568}]
I20260619 11:05:04.694053 18890 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 8edd097397094c70bb45fa9f95b9fcee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:04.696973 18904 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.697084 18904 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 491d97948e6c4dcd9f5247f1a6a8073c: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:04.698704 18888 tablet_replica.cc:439] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } } }
I20260619 11:05:04.698815 18888 tablet_replica.cc:442] T 34e358ac70b4426b840bf5f7c751c56d P 21b90faf51224e6aa6b1d377c066f71d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:04.700121 18911 mvcc.cc:205] Tried to move back new op lower bound from 7298527660786925568 to 7298527660680790016. Current Snapshot: MvccSnapshot[applied={T|T < 7298527660786925568}]
I20260619 11:05:04.704402 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.700287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.16.61.2:34473)
0619 11:05:04.700394 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.700399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.700442 (+ 43us) server_negotiation.cc:407] Connection header received
0619 11:05:04.700626 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.700629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.700685 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.700796 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:04.701093 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.701861 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.702786 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.703125 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.703236 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.703507 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.704004 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.704132 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.704218 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.704271 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:05:04.719969 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.715851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:35209 (local address 127.16.61.3:42387)
0619 11:05:04.715990 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.715994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.716010 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:05:04.716056 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.716059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.716106 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.716183 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:04.716561 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.717409 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.718665 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.719009 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.719110 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.719468 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.719549 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.719620 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.719782 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.719836 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:05:04.720002 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.712057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42387 (local address 127.16.61.2:35209)
0619 11:05:04.712426 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.712442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.712530 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.716308 (+ 3778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.716311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.716319 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:04.716437 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.716444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.717521 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.717524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.718536 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.718546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.719148 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.719152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.719256 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.719306 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.719647 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.719653 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.719757 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.719837 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
I20260619 11:05:04.727007 18533 raft_consensus.cc:1261] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Refusing update from remote peer 8edd097397094c70bb45fa9f95b9fcee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:04.727353 18813 raft_consensus.cc:1261] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Refusing update from remote peer 8edd097397094c70bb45fa9f95b9fcee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:04.727559 18892 consensus_queue.cc:1035] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:04.727808 18890 consensus_queue.cc:1035] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [LEADER]: Connected to new peer: Peer: permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:04.749639 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.745413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34473 (local address 127.16.61.3:45527)
0619 11:05:04.745721 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.745734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.745865 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.746192 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.746195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.746204 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:04.746328 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.746335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.747373 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.747376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.748400 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.748410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.748839 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.748843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.748937 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.748989 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.749340 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.749349 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.749416 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.749492 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260619 11:05:04.749624 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.745485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:45527 (local address 127.16.61.2:34473)
0619 11:05:04.745603 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.745607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.745826 (+ 219us) server_negotiation.cc:407] Connection header received
0619 11:05:04.745965 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.745968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.746015 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.746088 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:04.746446 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.747232 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.748519 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.748730 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.748813 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.749134 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.749223 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.749335 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.749442 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.749487 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:05:04.776254 18376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:04.776453 18376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:04.779147 18376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: Sending DeleteTablet for 3 replicas of tablet 2ae54cbbf6a74ef1b638a5029508477a
I20260619 11:05:04.779875 18793 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae54cbbf6a74ef1b638a5029508477a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:04 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:35952
I20260619 11:05:04.780066 18513 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae54cbbf6a74ef1b638a5029508477a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:04 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:55452
I20260619 11:05:04.780535 18931 tablet_replica.cc:330] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: stopping tablet replica
I20260619 11:05:04.780813 18653 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ae54cbbf6a74ef1b638a5029508477a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:04 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:46946
I20260619 11:05:04.780901 18931 raft_consensus.cc:2229] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:04.781261 18931 raft_consensus.cc:2258] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:04.781996 18932 tablet_replica.cc:330] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: stopping tablet replica
I20260619 11:05:04.782037 18376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:05:04.782258 18932 raft_consensus.cc:2229] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 1 LEADER]: Raft consensus shutting down.
W20260619 11:05:04.782438 18376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:04.782605 18932 raft_consensus.cc:2258] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:04.783941 18932 ts_tablet_manager.cc:1905] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:04.785109 18931 ts_tablet_manager.cc:1905] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:04.787925 18513 tablet_service.cc:1467] Processing CreateTablet for tablet 754181ce7a8c4c648440c176edc79a37 (DEFAULT_TABLE table=test-table [id=0e6cb8b648364e6eba6dccfabff7a294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:04.787961 18793 tablet_service.cc:1467] Processing CreateTablet for tablet 754181ce7a8c4c648440c176edc79a37 (DEFAULT_TABLE table=test-table [id=0e6cb8b648364e6eba6dccfabff7a294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:04.788182 18793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754181ce7a8c4c648440c176edc79a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.788182 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754181ce7a8c4c648440c176edc79a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.788715 18932 ts_tablet_manager.cc:1918] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:04.788796 18932 log.cc:1196] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/2ae54cbbf6a74ef1b638a5029508477a
I20260619 11:05:04.789100 18932 ts_tablet_manager.cc:1939] T 2ae54cbbf6a74ef1b638a5029508477a P 8edd097397094c70bb45fa9f95b9fcee: Deleting consensus metadata
I20260619 11:05:04.790391 18882 tablet_bootstrap.cc:492] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: Bootstrap starting.
I20260619 11:05:04.790504 18884 tablet_bootstrap.cc:492] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: Bootstrap starting.
I20260619 11:05:04.791311 18882 tablet_bootstrap.cc:654] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.791311 18884 tablet_bootstrap.cc:654] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.792018 18363 catalog_manager.cc:4841] TS 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473): tablet 2ae54cbbf6a74ef1b638a5029508477a (table test-table [id=4f9af1c8b7f44bb4a5e2ffdf5da3166b]) successfully deleted
I20260619 11:05:04.792702 18882 tablet_bootstrap.cc:492] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: No bootstrap required, opened a new log
I20260619 11:05:04.792742 18653 tablet_service.cc:1467] Processing CreateTablet for tablet 754181ce7a8c4c648440c176edc79a37 (DEFAULT_TABLE table=test-table [id=0e6cb8b648364e6eba6dccfabff7a294]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:04.792788 18882 ts_tablet_manager.cc:1397] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:04.792989 18653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 754181ce7a8c4c648440c176edc79a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:04.793232 18882 raft_consensus.cc:348] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.793361 18882 raft_consensus.cc:374] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.793401 18882 raft_consensus.cc:729] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b90faf51224e6aa6b1d377c066f71d, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.793517 18882 consensus_queue.cc:260] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.793834 18882 ts_tablet_manager.cc:1428] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:04.793963 18931 ts_tablet_manager.cc:1918] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:04.794095 18931 log.cc:1196] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/2ae54cbbf6a74ef1b638a5029508477a
I20260619 11:05:04.794476 18931 ts_tablet_manager.cc:1939] T 2ae54cbbf6a74ef1b638a5029508477a P 21b90faf51224e6aa6b1d377c066f71d: Deleting consensus metadata
I20260619 11:05:04.795078 18883 tablet_bootstrap.cc:492] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: Bootstrap starting.
I20260619 11:05:04.795804 18360 catalog_manager.cc:4841] TS 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479): tablet 2ae54cbbf6a74ef1b638a5029508477a (table test-table [id=4f9af1c8b7f44bb4a5e2ffdf5da3166b]) successfully deleted
I20260619 11:05:04.796042 18883 tablet_bootstrap.cc:654] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:04.796366 18884 tablet_bootstrap.cc:492] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: No bootstrap required, opened a new log
I20260619 11:05:04.796480 18884 ts_tablet_manager.cc:1397] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 11:05:04.796994 18884 raft_consensus.cc:348] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.797122 18884 raft_consensus.cc:374] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.797154 18884 raft_consensus.cc:729] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491d97948e6c4dcd9f5247f1a6a8073c, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.797209 18883 tablet_bootstrap.cc:492] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: No bootstrap required, opened a new log
I20260619 11:05:04.797276 18883 ts_tablet_manager.cc:1397] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:04.797307 18884 consensus_queue.cc:260] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.797581 18884 ts_tablet_manager.cc:1428] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 11:05:04.797751 18883 raft_consensus.cc:348] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.797863 18883 raft_consensus.cc:374] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:04.797896 18883 raft_consensus.cc:729] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edd097397094c70bb45fa9f95b9fcee, State: Initialized, Role: FOLLOWER
I20260619 11:05:04.798125 18930 tablet_replica.cc:330] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: stopping tablet replica
I20260619 11:05:04.798440 18930 raft_consensus.cc:2229] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:04.798470 18883 consensus_queue.cc:260] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.798609 18930 raft_consensus.cc:2258] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:04.798770 18883 ts_tablet_manager.cc:1428] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:04.799947 18930 ts_tablet_manager.cc:1905] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:04.802726 18930 ts_tablet_manager.cc:1918] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:04.802821 18930 log.cc:1196] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/2ae54cbbf6a74ef1b638a5029508477a
I20260619 11:05:04.803149 18930 ts_tablet_manager.cc:1939] T 2ae54cbbf6a74ef1b638a5029508477a P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting consensus metadata
I20260619 11:05:04.804324 18362 catalog_manager.cc:4841] TS 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387): tablet 2ae54cbbf6a74ef1b638a5029508477a (table test-table [id=4f9af1c8b7f44bb4a5e2ffdf5da3166b]) successfully deleted
I20260619 11:05:04.914553 18904 raft_consensus.cc:482] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:04.914726 18904 raft_consensus.cc:504] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.915179 18904 leader_election.cc:288] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479), 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473)
I20260619 11:05:04.915622 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754181ce7a8c4c648440c176edc79a37" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b90faf51224e6aa6b1d377c066f71d" is_pre_election: true
I20260619 11:05:04.915622 18673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754181ce7a8c4c648440c176edc79a37" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd097397094c70bb45fa9f95b9fcee" is_pre_election: true
I20260619 11:05:04.915792 18533 raft_consensus.cc:2454] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 0.
I20260619 11:05:04.915807 18673 raft_consensus.cc:2454] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 0.
I20260619 11:05:04.916141 18744 leader_election.cc:302] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b90faf51224e6aa6b1d377c066f71d, 491d97948e6c4dcd9f5247f1a6a8073c; no voters:
I20260619 11:05:04.916299 18904 raft_consensus.cc:2781] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:04.916378 18904 raft_consensus.cc:482] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:04.916420 18904 raft_consensus.cc:3037] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.917713 18904 raft_consensus.cc:504] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.918067 18904 leader_election.cc:288] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 election: Requested vote from peers 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479), 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473)
I20260619 11:05:04.918413 18533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754181ce7a8c4c648440c176edc79a37" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b90faf51224e6aa6b1d377c066f71d"
I20260619 11:05:04.918452 18673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "754181ce7a8c4c648440c176edc79a37" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd097397094c70bb45fa9f95b9fcee"
I20260619 11:05:04.918534 18533 raft_consensus.cc:3037] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.918578 18673 raft_consensus.cc:3037] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:04.919479 18673 raft_consensus.cc:2454] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 1.
I20260619 11:05:04.919585 18533 raft_consensus.cc:2454] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 1.
I20260619 11:05:04.919806 18747 leader_election.cc:302] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491d97948e6c4dcd9f5247f1a6a8073c, 8edd097397094c70bb45fa9f95b9fcee; no voters:
I20260619 11:05:04.919993 18904 raft_consensus.cc:2781] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:04.920087 18904 raft_consensus.cc:686] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 LEADER]: Becoming Leader. State: Replica: 491d97948e6c4dcd9f5247f1a6a8073c, State: Running, Role: LEADER
I20260619 11:05:04.920214 18904 consensus_queue.cc:237] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:04.921950 18376 catalog_manager.cc:5495] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c reported cstate change: term changed from 0 to 1, leader changed from <none> to 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3). New cstate: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } health_report { overall_health: HEALTHY } } }
I20260619 11:05:04.959774 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.956635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.16.61.3:42387)
0619 11:05:04.956769 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.956773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.956822 (+ 49us) server_negotiation.cc:407] Connection header received
0619 11:05:04.956988 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.956991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.957041 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.957130 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:04.957379 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.957877 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.958707 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.958937 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.959060 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.959314 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.959387 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.959473 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.959565 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.959612 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:05:04.968158 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.965126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42387 (local address 127.16.61.3:54423)
0619 11:05:04.965286 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:04.965299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:04.965400 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:04.965614 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:04.965617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:04.965622 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:04.965769 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.965774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.966474 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.966477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.967030 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:04.967036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.967467 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.967470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:04.967543 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.967575 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.967877 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.967882 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:05:04.967940 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 11:05:04.968033 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":34}
I20260619 11:05:04.968158 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:04.965296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:54423 (local address 127.16.61.3:42387)
0619 11:05:04.965400 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:04.965405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:04.965417 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:04.965481 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:04.965486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:04.965526 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:04.965610 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:04.965887 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.966360 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.967143 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:04.967360 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:04.967470 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:04.967710 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:04.967777 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:04.967860 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:04.967947 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 11:05:04.968012 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":50}
I20260619 11:05:04.973645 18533 raft_consensus.cc:1261] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Refusing update from remote peer 491d97948e6c4dcd9f5247f1a6a8073c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:04.973760 18673 raft_consensus.cc:1261] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Refusing update from remote peer 491d97948e6c4dcd9f5247f1a6a8073c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:04.974144 18904 consensus_queue.cc:1035] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:04.974395 18889 consensus_queue.cc:1035] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:05:04.988880 18909 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 11:05:06.541358 18374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:06.541535 18374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:06.543846 18374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: Sending DeleteTablet for 3 replicas of tablet 754181ce7a8c4c648440c176edc79a37
I20260619 11:05:06.544320 18653 tablet_service.cc:1514] Processing DeleteTablet for tablet 754181ce7a8c4c648440c176edc79a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:06 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:46946
I20260619 11:05:06.544430 18793 tablet_service.cc:1514] Processing DeleteTablet for tablet 754181ce7a8c4c648440c176edc79a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:06 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:35952
I20260619 11:05:06.544430 18513 tablet_service.cc:1514] Processing DeleteTablet for tablet 754181ce7a8c4c648440c176edc79a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:06 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:55452
I20260619 11:05:06.544674 18960 tablet_replica.cc:330] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: stopping tablet replica
I20260619 11:05:06.544731 18962 tablet_replica.cc:330] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: stopping tablet replica
I20260619 11:05:06.544803 18960 raft_consensus.cc:2229] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:06.544857 18962 raft_consensus.cc:2229] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:06.544909 18960 raft_consensus.cc:2258] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:06.544970 18962 raft_consensus.cc:2258] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:06.545404 18961 tablet_replica.cc:330] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: stopping tablet replica
I20260619 11:05:06.545521 18960 ts_tablet_manager.cc:1905] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:06.545522 18961 raft_consensus.cc:2229] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:06.545796 18961 raft_consensus.cc:2258] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:06.546317 18961 ts_tablet_manager.cc:1905] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:06.548341 18960 ts_tablet_manager.cc:1918] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:05:06.548353 18962 ts_tablet_manager.cc:1905] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:06.548426 18960 log.cc:1196] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/754181ce7a8c4c648440c176edc79a37
I20260619 11:05:06.548694 18960 ts_tablet_manager.cc:1939] T 754181ce7a8c4c648440c176edc79a37 P 8edd097397094c70bb45fa9f95b9fcee: Deleting consensus metadata
I20260619 11:05:06.549024 18961 ts_tablet_manager.cc:1918] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:05:06.549104 18961 log.cc:1196] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/754181ce7a8c4c648440c176edc79a37
I20260619 11:05:06.549397 18961 ts_tablet_manager.cc:1939] T 754181ce7a8c4c648440c176edc79a37 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting consensus metadata
I20260619 11:05:06.549417 18363 catalog_manager.cc:4841] TS 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473): tablet 754181ce7a8c4c648440c176edc79a37 (table test-table [id=0e6cb8b648364e6eba6dccfabff7a294]) successfully deleted
I20260619 11:05:06.550014 18362 catalog_manager.cc:4841] TS 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387): tablet 754181ce7a8c4c648440c176edc79a37 (table test-table [id=0e6cb8b648364e6eba6dccfabff7a294]) successfully deleted
I20260619 11:05:06.551208 18962 ts_tablet_manager.cc:1918] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:05:06.551316 18962 log.cc:1196] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/754181ce7a8c4c648440c176edc79a37
I20260619 11:05:06.551561 18962 ts_tablet_manager.cc:1939] T 754181ce7a8c4c648440c176edc79a37 P 21b90faf51224e6aa6b1d377c066f71d: Deleting consensus metadata
I20260619 11:05:06.552207 18360 catalog_manager.cc:4841] TS 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479): tablet 754181ce7a8c4c648440c176edc79a37 (table test-table [id=0e6cb8b648364e6eba6dccfabff7a294]) successfully deleted
Jun 19 11:05:06 dist-test-slave-8vcz krb5kdc[18331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867106, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:05:06 dist-test-slave-8vcz krb5kdc[18331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867106, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:06.577253 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.563529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.16.61.62:35307)
0619 11:05:06.563851 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:06.563858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:06.563872 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:05:06.563999 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:06.564002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:06.564044 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:06.564123 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:06.564851 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.565504 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.566640 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.566868 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.566962 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.573640 (+ 6678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:06.573643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:06.573645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:06.573673 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:06.575262 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:06.575907 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:06.575910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:06.575911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:06.575992 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:06.576271 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:06.576273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:06.576274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:06.576498 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:06.576628 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:06.576921 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 11:05:06.577025 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":153,"threads_started":1}
I20260619 11:05:06.580353 18374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:06.581061 18374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:06.585973 18513 tablet_service.cc:1467] Processing CreateTablet for tablet c24ffbd4fa2748de8303dae8d4e600c0 (DEFAULT_TABLE table=test-table [id=48f8d5b1019b43b587da06476efb1e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:06.586051 18653 tablet_service.cc:1467] Processing CreateTablet for tablet c24ffbd4fa2748de8303dae8d4e600c0 (DEFAULT_TABLE table=test-table [id=48f8d5b1019b43b587da06476efb1e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:06.586158 18793 tablet_service.cc:1467] Processing CreateTablet for tablet c24ffbd4fa2748de8303dae8d4e600c0 (DEFAULT_TABLE table=test-table [id=48f8d5b1019b43b587da06476efb1e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:06.586292 18653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24ffbd4fa2748de8303dae8d4e600c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:06.586292 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24ffbd4fa2748de8303dae8d4e600c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:06.586448 18793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c24ffbd4fa2748de8303dae8d4e600c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:06.588809 18973 tablet_bootstrap.cc:492] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: Bootstrap starting.
I20260619 11:05:06.588877 18971 tablet_bootstrap.cc:492] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: Bootstrap starting.
I20260619 11:05:06.589637 18973 tablet_bootstrap.cc:654] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:06.589636 18971 tablet_bootstrap.cc:654] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:06.590296 18972 tablet_bootstrap.cc:492] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: Bootstrap starting.
I20260619 11:05:06.590798 18971 tablet_bootstrap.cc:492] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: No bootstrap required, opened a new log
I20260619 11:05:06.590868 18973 tablet_bootstrap.cc:492] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: No bootstrap required, opened a new log
I20260619 11:05:06.590909 18971 ts_tablet_manager.cc:1397] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:06.590950 18973 ts_tablet_manager.cc:1397] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:06.591248 18972 tablet_bootstrap.cc:654] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:06.591336 18971 raft_consensus.cc:348] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.591370 18973 raft_consensus.cc:348] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.591454 18971 raft_consensus.cc:374] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:06.591490 18973 raft_consensus.cc:374] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:06.591493 18971 raft_consensus.cc:729] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491d97948e6c4dcd9f5247f1a6a8073c, State: Initialized, Role: FOLLOWER
I20260619 11:05:06.591532 18973 raft_consensus.cc:729] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edd097397094c70bb45fa9f95b9fcee, State: Initialized, Role: FOLLOWER
I20260619 11:05:06.591645 18971 consensus_queue.cc:260] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.591724 18973 consensus_queue.cc:260] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.592103 18973 ts_tablet_manager.cc:1428] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:06.592430 18971 ts_tablet_manager.cc:1428] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:05:06.592661 18972 tablet_bootstrap.cc:492] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: No bootstrap required, opened a new log
I20260619 11:05:06.592785 18972 ts_tablet_manager.cc:1397] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:05:06.593225 18972 raft_consensus.cc:348] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.593331 18972 raft_consensus.cc:374] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:06.593380 18972 raft_consensus.cc:729] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b90faf51224e6aa6b1d377c066f71d, State: Initialized, Role: FOLLOWER
I20260619 11:05:06.593513 18972 consensus_queue.cc:260] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.593909 18972 ts_tablet_manager.cc:1428] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:06.670338 18978 raft_consensus.cc:482] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:06.670522 18978 raft_consensus.cc:504] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.671731 18978 leader_election.cc:288] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473), 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387)
I20260619 11:05:06.675743 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.672074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:54719 (local address 127.16.61.3:42387)
0619 11:05:06.672432 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:06.672442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:06.672457 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:05:06.672544 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:06.672547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:06.672595 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:06.672693 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:06.672993 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.673503 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.674579 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.674986 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.675106 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.675312 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:06.675515 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 11:05:06.675589 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20260619 11:05:06.675761 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.671953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42387 (local address 127.16.61.1:54719)
0619 11:05:06.672323 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:06.672339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:06.672446 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:06.672706 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:06.672709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:06.672732 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:06.672857 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:06.672863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.673614 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.673620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:06.674454 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:06.674462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.675111 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.675114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:06.675355 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.675391 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:05:06.675498 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:05:06.675573 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260619 11:05:06.676168 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.671953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34473 (local address 127.16.61.1:34533)
0619 11:05:06.672484 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:06.672497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:06.672726 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:06.673067 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:06.673070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:06.673085 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:06.673206 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:06.673212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.673968 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.673971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:06.674729 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:06.674737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.675641 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.675647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:06.675790 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.675849 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 11:05:06.675920 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:05:06.676006 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":35,"thread_start_us":119,"threads_started":1}
I20260619 11:05:06.676214 18813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24ffbd4fa2748de8303dae8d4e600c0" candidate_uuid: "21b90faf51224e6aa6b1d377c066f71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" is_pre_election: true
I20260619 11:05:06.676362 18813 raft_consensus.cc:2454] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b90faf51224e6aa6b1d377c066f71d in term 0.
I20260619 11:05:06.676489 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.672081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:34533 (local address 127.16.61.2:34473)
0619 11:05:06.672413 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:06.672423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:06.672591 (+ 168us) server_negotiation.cc:407] Connection header received
0619 11:05:06.672866 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:06.672869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:06.672952 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:06.673059 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:06.673323 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.673833 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.674862 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.675454 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.676189 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.676256 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:06.676293 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 11:05:06.676362 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":120,"threads_started":1}
I20260619 11:05:06.676785 18466 leader_election.cc:302] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b90faf51224e6aa6b1d377c066f71d, 491d97948e6c4dcd9f5247f1a6a8073c; no voters:
I20260619 11:05:06.676955 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24ffbd4fa2748de8303dae8d4e600c0" candidate_uuid: "21b90faf51224e6aa6b1d377c066f71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd097397094c70bb45fa9f95b9fcee" is_pre_election: true
I20260619 11:05:06.677083 18978 raft_consensus.cc:2781] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:06.677146 18672 raft_consensus.cc:2454] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21b90faf51224e6aa6b1d377c066f71d in term 0.
I20260619 11:05:06.677183 18978 raft_consensus.cc:482] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:06.677240 18978 raft_consensus.cc:3037] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:06.678218 18978 raft_consensus.cc:504] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.678596 18978 leader_election.cc:288] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [CANDIDATE]: Term 1 election: Requested vote from peers 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473), 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387)
I20260619 11:05:06.679163 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24ffbd4fa2748de8303dae8d4e600c0" candidate_uuid: "21b90faf51224e6aa6b1d377c066f71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd097397094c70bb45fa9f95b9fcee"
I20260619 11:05:06.679163 18813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c24ffbd4fa2748de8303dae8d4e600c0" candidate_uuid: "21b90faf51224e6aa6b1d377c066f71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "491d97948e6c4dcd9f5247f1a6a8073c"
I20260619 11:05:06.679293 18813 raft_consensus.cc:3037] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:06.679293 18672 raft_consensus.cc:3037] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:06.680457 18672 raft_consensus.cc:2454] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b90faf51224e6aa6b1d377c066f71d in term 1.
I20260619 11:05:06.680461 18813 raft_consensus.cc:2454] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21b90faf51224e6aa6b1d377c066f71d in term 1.
I20260619 11:05:06.680780 18466 leader_election.cc:302] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b90faf51224e6aa6b1d377c066f71d, 491d97948e6c4dcd9f5247f1a6a8073c; no voters:
I20260619 11:05:06.680971 18978 raft_consensus.cc:2781] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:06.681267 18978 raft_consensus.cc:686] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 1 LEADER]: Becoming Leader. State: Replica: 21b90faf51224e6aa6b1d377c066f71d, State: Running, Role: LEADER
I20260619 11:05:06.681591 18978 consensus_queue.cc:237] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.684028 18375 catalog_manager.cc:5495] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d reported cstate change: term changed from 0 to 1, leader changed from <none> to 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1). New cstate: current_term: 1 leader_uuid: "21b90faf51224e6aa6b1d377c066f71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:06.722975 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.718600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.16.61.1:39479)
0619 11:05:06.718879 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:06.718887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:06.718900 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:06.718982 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:06.718985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:06.719048 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:06.719171 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:06.719452 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.720250 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.721214 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.721578 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.721685 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.721965 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:06.722540 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:06.722649 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:06.722750 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 11:05:06.722803 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":116,"threads_started":1}
I20260619 11:05:06.733085 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.730070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:56953 (local address 127.16.61.3:42387)
0619 11:05:06.730198 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:06.730202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:06.730229 (+ 27us) server_negotiation.cc:407] Connection header received
0619 11:05:06.730334 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:06.730336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:06.730398 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:06.730463 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:06.730685 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.731312 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.732224 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.732437 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.732512 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.732745 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:06.732804 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:06.732855 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:06.732944 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 11:05:06.732985 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:05:06.733103 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.729861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42387 (local address 127.16.61.1:56953)
0619 11:05:06.730173 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:06.730186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:06.730266 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:06.730491 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:06.730493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:06.730499 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:06.730596 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:06.730600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.731413 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.731416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:06.732126 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:06.732134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.732533 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.732535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:06.732602 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.732640 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:06.732883 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:06.732888 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:05:06.732937 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 11:05:06.732994 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
I20260619 11:05:06.738981 18813 raft_consensus.cc:1261] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Refusing update from remote peer 21b90faf51224e6aa6b1d377c066f71d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:06.738969 18672 raft_consensus.cc:1261] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Refusing update from remote peer 21b90faf51224e6aa6b1d377c066f71d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:06.739470 18978 consensus_queue.cc:1035] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:06.739704 18983 consensus_queue.cc:1035] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:06.756031 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.752426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:39479 (local address 127.16.61.3:40909)
0619 11:05:06.752743 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:06.752756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:06.752873 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:06.753124 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:06.753126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:06.753134 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:06.753222 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:06.753226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.753872 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.753874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:06.754666 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:06.754676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.755187 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.755191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:06.755276 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.755300 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:06.755729 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:06.755741 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 11:05:06.755832 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 11:05:06.755904 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260619 11:05:06.756100 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.752480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:40909 (local address 127.16.61.1:39479)
0619 11:05:06.752577 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:06.752582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:06.752833 (+ 251us) server_negotiation.cc:407] Connection header received
0619 11:05:06.752954 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:06.752957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:06.753000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:06.753112 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:06.753294 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.753774 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.754786 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.755068 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.755164 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.755458 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:06.755540 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:06.755611 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:06.755954 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 11:05:06.756001 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 11:05:06.776430 18375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:06.776611 18375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:06.778739 18375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: Sending DeleteTablet for 3 replicas of tablet c24ffbd4fa2748de8303dae8d4e600c0
I20260619 11:05:06.779435 18653 tablet_service.cc:1514] Processing DeleteTablet for tablet c24ffbd4fa2748de8303dae8d4e600c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:06 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:46946
I20260619 11:05:06.779457 18793 tablet_service.cc:1514] Processing DeleteTablet for tablet c24ffbd4fa2748de8303dae8d4e600c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:06 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:35952
I20260619 11:05:06.779485 18513 tablet_service.cc:1514] Processing DeleteTablet for tablet c24ffbd4fa2748de8303dae8d4e600c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:06 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:55452
I20260619 11:05:06.779646 18960 tablet_replica.cc:330] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: stopping tablet replica
I20260619 11:05:06.779639 18961 tablet_replica.cc:330] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: stopping tablet replica
I20260619 11:05:06.779775 18961 raft_consensus.cc:2229] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:06.779776 18960 raft_consensus.cc:2229] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:06.779888 18960 raft_consensus.cc:2258] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:06.779888 18961 raft_consensus.cc:2258] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:06.780051 18962 tablet_replica.cc:330] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: stopping tablet replica
I20260619 11:05:06.780151 18962 raft_consensus.cc:2229] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:06.780452 18962 raft_consensus.cc:2258] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:06.780792 18961 ts_tablet_manager.cc:1905] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:06.781347 18960 ts_tablet_manager.cc:1905] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:06.783551 18961 ts_tablet_manager.cc:1918] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:06.783633 18961 log.cc:1196] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/c24ffbd4fa2748de8303dae8d4e600c0
I20260619 11:05:06.783932 18961 ts_tablet_manager.cc:1939] T c24ffbd4fa2748de8303dae8d4e600c0 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting consensus metadata
I20260619 11:05:06.784112 18962 ts_tablet_manager.cc:1905] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:06.785073 18960 ts_tablet_manager.cc:1918] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:06.785151 18960 log.cc:1196] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/c24ffbd4fa2748de8303dae8d4e600c0
I20260619 11:05:06.785423 18960 ts_tablet_manager.cc:1939] T c24ffbd4fa2748de8303dae8d4e600c0 P 8edd097397094c70bb45fa9f95b9fcee: Deleting consensus metadata
I20260619 11:05:06.786121 18363 catalog_manager.cc:4841] TS 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473): tablet c24ffbd4fa2748de8303dae8d4e600c0 (table test-table [id=48f8d5b1019b43b587da06476efb1e9d]) successfully deleted
I20260619 11:05:06.786197 18362 catalog_manager.cc:4841] TS 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387): tablet c24ffbd4fa2748de8303dae8d4e600c0 (table test-table [id=48f8d5b1019b43b587da06476efb1e9d]) successfully deleted
I20260619 11:05:06.786396 18962 ts_tablet_manager.cc:1918] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:06.786099 18375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:05:06.786497 18962 log.cc:1196] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/c24ffbd4fa2748de8303dae8d4e600c0
I20260619 11:05:06.786754 18962 ts_tablet_manager.cc:1939] T c24ffbd4fa2748de8303dae8d4e600c0 P 21b90faf51224e6aa6b1d377c066f71d: Deleting consensus metadata
W20260619 11:05:06.786756 18375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:06.787393 18360 catalog_manager.cc:4841] TS 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479): tablet c24ffbd4fa2748de8303dae8d4e600c0 (table test-table [id=48f8d5b1019b43b587da06476efb1e9d]) successfully deleted
I20260619 11:05:06.791247 18513 tablet_service.cc:1467] Processing CreateTablet for tablet 4749b8dce8e6448da2990378f8d8feb4 (DEFAULT_TABLE table=test-table [id=2fc671b937a84ad9b2aa97ad7cee129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:06.791531 18513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4749b8dce8e6448da2990378f8d8feb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:06.791508 18793 tablet_service.cc:1467] Processing CreateTablet for tablet 4749b8dce8e6448da2990378f8d8feb4 (DEFAULT_TABLE table=test-table [id=2fc671b937a84ad9b2aa97ad7cee129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:06.791611 18653 tablet_service.cc:1467] Processing CreateTablet for tablet 4749b8dce8e6448da2990378f8d8feb4 (DEFAULT_TABLE table=test-table [id=2fc671b937a84ad9b2aa97ad7cee129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:06.791770 18793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4749b8dce8e6448da2990378f8d8feb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:06.791867 18653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4749b8dce8e6448da2990378f8d8feb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:06.793790 18972 tablet_bootstrap.cc:492] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d: Bootstrap starting.
I20260619 11:05:06.793953 18971 tablet_bootstrap.cc:492] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: Bootstrap starting.
I20260619 11:05:06.794035 18973 tablet_bootstrap.cc:492] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: Bootstrap starting.
I20260619 11:05:06.794744 18972 tablet_bootstrap.cc:654] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:06.794744 18971 tablet_bootstrap.cc:654] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:06.794839 18973 tablet_bootstrap.cc:654] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:06.796123 18972 tablet_bootstrap.cc:492] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d: No bootstrap required, opened a new log
I20260619 11:05:06.796227 18972 ts_tablet_manager.cc:1397] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:06.796238 18971 tablet_bootstrap.cc:492] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: No bootstrap required, opened a new log
I20260619 11:05:06.796307 18971 ts_tablet_manager.cc:1397] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 11:05:06.796693 18972 raft_consensus.cc:348] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.796693 18971 raft_consensus.cc:348] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.796808 18971 raft_consensus.cc:374] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:06.796808 18972 raft_consensus.cc:374] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:06.796841 18972 raft_consensus.cc:729] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b90faf51224e6aa6b1d377c066f71d, State: Initialized, Role: FOLLOWER
I20260619 11:05:06.796841 18971 raft_consensus.cc:729] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 491d97948e6c4dcd9f5247f1a6a8073c, State: Initialized, Role: FOLLOWER
I20260619 11:05:06.796918 18973 tablet_bootstrap.cc:492] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: No bootstrap required, opened a new log
I20260619 11:05:06.796964 18972 consensus_queue.cc:260] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.796964 18971 consensus_queue.cc:260] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.797012 18973 ts_tablet_manager.cc:1397] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:05:06.797212 18971 ts_tablet_manager.cc:1428] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:05:06.797212 18972 ts_tablet_manager.cc:1428] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:06.797377 18973 raft_consensus.cc:348] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.797493 18973 raft_consensus.cc:374] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:06.797524 18973 raft_consensus.cc:729] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edd097397094c70bb45fa9f95b9fcee, State: Initialized, Role: FOLLOWER
I20260619 11:05:06.797624 18973 consensus_queue.cc:260] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.797851 18973 ts_tablet_manager.cc:1428] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:06.821266 18916 raft_consensus.cc:482] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:06.821414 18916 raft_consensus.cc:504] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.821877 18916 leader_election.cc:288] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479), 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473)
I20260619 11:05:06.822268 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4749b8dce8e6448da2990378f8d8feb4" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd097397094c70bb45fa9f95b9fcee" is_pre_election: true
I20260619 11:05:06.822417 18672 raft_consensus.cc:2454] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 0.
I20260619 11:05:06.822438 18532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4749b8dce8e6448da2990378f8d8feb4" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b90faf51224e6aa6b1d377c066f71d" is_pre_election: true
I20260619 11:05:06.822597 18532 raft_consensus.cc:2454] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 0.
I20260619 11:05:06.822707 18747 leader_election.cc:302] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491d97948e6c4dcd9f5247f1a6a8073c, 8edd097397094c70bb45fa9f95b9fcee; no voters:
I20260619 11:05:06.822896 18916 raft_consensus.cc:2781] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:06.822957 18916 raft_consensus.cc:482] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:06.823002 18916 raft_consensus.cc:3037] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:06.824091 18916 raft_consensus.cc:504] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.824460 18916 leader_election.cc:288] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 election: Requested vote from peers 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479), 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473)
I20260619 11:05:06.824784 18672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4749b8dce8e6448da2990378f8d8feb4" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edd097397094c70bb45fa9f95b9fcee"
I20260619 11:05:06.824796 18532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4749b8dce8e6448da2990378f8d8feb4" candidate_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b90faf51224e6aa6b1d377c066f71d"
I20260619 11:05:06.824914 18532 raft_consensus.cc:3037] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:06.824919 18672 raft_consensus.cc:3037] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:06.825970 18672 raft_consensus.cc:2454] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 1.
I20260619 11:05:06.826129 18532 raft_consensus.cc:2454] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 491d97948e6c4dcd9f5247f1a6a8073c in term 1.
I20260619 11:05:06.826247 18747 leader_election.cc:302] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 491d97948e6c4dcd9f5247f1a6a8073c, 8edd097397094c70bb45fa9f95b9fcee; no voters:
I20260619 11:05:06.826438 18916 raft_consensus.cc:2781] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:06.826537 18916 raft_consensus.cc:686] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 LEADER]: Becoming Leader. State: Replica: 491d97948e6c4dcd9f5247f1a6a8073c, State: Running, Role: LEADER
I20260619 11:05:06.826671 18916 consensus_queue.cc:237] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } }
I20260619 11:05:06.828279 18376 catalog_manager.cc:5495] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c reported cstate change: term changed from 0 to 1, leader changed from <none> to 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3). New cstate: current_term: 1 leader_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "491d97948e6c4dcd9f5247f1a6a8073c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42387 } health_report { overall_health: HEALTHY } } }
I20260619 11:05:06.848352 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:06.845003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.16.61.3:42387)
0619 11:05:06.845111 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:06.845116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:06.845128 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:06.845274 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:06.845276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:06.845315 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:06.845388 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:06.845634 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.846118 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.847259 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:06.847500 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:06.847578 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:06.847886 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:06.847953 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:06.848004 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:06.848123 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 11:05:06.848160 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:05:06.853983 18672 raft_consensus.cc:1261] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Refusing update from remote peer 491d97948e6c4dcd9f5247f1a6a8073c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:06.854101 18533 raft_consensus.cc:1261] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d [term 1 FOLLOWER]: Refusing update from remote peer 491d97948e6c4dcd9f5247f1a6a8073c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:06.854576 18916 consensus_queue.cc:1035] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8edd097397094c70bb45fa9f95b9fcee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:06.854743 18948 consensus_queue.cc:1035] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b90faf51224e6aa6b1d377c066f71d" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 39479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:05:06.868867 18941 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 11:05:08.418748 18376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:08.418897 18376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:08.421200 18376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 148eff810f04433882fe37bbfceea5a6: Sending DeleteTablet for 3 replicas of tablet 4749b8dce8e6448da2990378f8d8feb4
I20260619 11:05:08.421780 18653 tablet_service.cc:1514] Processing DeleteTablet for tablet 4749b8dce8e6448da2990378f8d8feb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:08 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:46946
I20260619 11:05:08.421829 18513 tablet_service.cc:1514] Processing DeleteTablet for tablet 4749b8dce8e6448da2990378f8d8feb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:08 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:55452
I20260619 11:05:08.422112 19023 tablet_replica.cc:330] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: stopping tablet replica
I20260619 11:05:08.422184 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 18448
I20260619 11:05:08.422195 19024 tablet_replica.cc:330] T 4749b8dce8e6448da2990378f8d8feb4 P 21b90faf51224e6aa6b1d377c066f71d: stopping tablet replica
I20260619 11:05:08.422315 18793 tablet_service.cc:1514] Processing DeleteTablet for tablet 4749b8dce8e6448da2990378f8d8feb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:08 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:35952
I20260619 11:05:08.422809 19023 raft_consensus.cc:2229] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:08.422902 19025 tablet_replica.cc:330] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: stopping tablet replica
I20260619 11:05:08.422930 19023 raft_consensus.cc:2258] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:08.423008 19025 raft_consensus.cc:2229] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:08.423177 19025 raft_consensus.cc:2258] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:08.423569 19023 ts_tablet_manager.cc:1905] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:08.424940 19025 ts_tablet_manager.cc:1905] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:08.426345 19023 ts_tablet_manager.cc:1918] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:05:08.426424 19023 log.cc:1196] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/4749b8dce8e6448da2990378f8d8feb4
I20260619 11:05:08.426731 19023 ts_tablet_manager.cc:1939] T 4749b8dce8e6448da2990378f8d8feb4 P 8edd097397094c70bb45fa9f95b9fcee: Deleting consensus metadata
I20260619 11:05:08.427456 18363 catalog_manager.cc:4841] TS 8edd097397094c70bb45fa9f95b9fcee (127.16.61.2:34473): tablet 4749b8dce8e6448da2990378f8d8feb4 (table test-table [id=2fc671b937a84ad9b2aa97ad7cee129c]) successfully deleted
I20260619 11:05:08.427553 19025 ts_tablet_manager.cc:1918] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 11:05:08.427618 19025 log.cc:1196] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/4749b8dce8e6448da2990378f8d8feb4
I20260619 11:05:08.427942 19025 ts_tablet_manager.cc:1939] T 4749b8dce8e6448da2990378f8d8feb4 P 491d97948e6c4dcd9f5247f1a6a8073c: Deleting consensus metadata
I20260619 11:05:08.428541 18362 catalog_manager.cc:4841] TS 491d97948e6c4dcd9f5247f1a6a8073c (127.16.61.3:42387): tablet 4749b8dce8e6448da2990378f8d8feb4 (table test-table [id=2fc671b937a84ad9b2aa97ad7cee129c]) successfully deleted
W20260619 11:05:08.430627 18360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:39479 (error 108)
I20260619 11:05:08.430778 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 18588
I20260619 11:05:08.432802 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:08.432422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:39479 (local address 127.0.0.1:55476)
0619 11:05:08.432657 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:08.432722 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:39479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":98,"threads_started":1}
W20260619 11:05:08.432940 18360 catalog_manager.cc:4584] TS 21b90faf51224e6aa6b1d377c066f71d (127.16.61.1:39479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4749b8dce8e6448da2990378f8d8feb4: Network error: Client connection negotiation failed: client connection to 127.16.61.1:39479: connect: Connection refused (error 111)
I20260619 11:05:08.438212 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 18728
I20260619 11:05:08.446360 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 18344
2026-06-19T11:05:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6372 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:08 dist-test-slave-8vcz krb5kdc[19030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:08 dist-test-slave-8vcz krb5kdc[19030](info): set up 2 sockets
Jun 19 11:05:08 dist-test-slave-8vcz krb5kdc[19030](info): commencing operation
krb5kdc: starting...
W20260619 11:05:09.487638 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:09 dist-test-slave-8vcz krb5kdc[19030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:09.608633 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35321
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:35321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:09.716892 19043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:09.717141 19043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:09.717178 19043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:09.721131 19043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:09.721187 19043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:09.721207 19043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:09.721225 19043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:09.721239 19043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:09.726336 19043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:35321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:09.727344 19043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:09.728370 19043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:09.736966 19051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:09.736966 19048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:09.736979 19049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:09.737033 19043 server_base.cc:1034] running on GCE node
I20260619 11:05:09.737902 19043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:09.738996 19043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:09.740190 19043 hybrid_clock.cc:648] HybridClock initialized: now 1781867109740151 us; error 78 us; skew 500 ppm
Jun 19 11:05:09 dist-test-slave-8vcz krb5kdc[19030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867109, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:09.742871 19043 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:09.743784 19043 webserver.cc:460] Webserver started at http://127.16.61.62:34221/ using document root <none> and password file <none>
I20260619 11:05:09.744341 19043 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:09.744402 19043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:09.744618 19043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:09.746596 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "eccbf7179e414a4a8cafc36f07ed9b18"
format_stamp: "Formatted at 2026-06-19 11:05:09 on dist-test-slave-8vcz"
server_key: "1245a1286b46ed50663204e90dfdbd02"
server_key_iv: "cd0a02d5aabc159f64d2a2b6bf3b3d9c"
server_key_version: "encryptionkey@0"
I20260619 11:05:09.747164 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "eccbf7179e414a4a8cafc36f07ed9b18"
format_stamp: "Formatted at 2026-06-19 11:05:09 on dist-test-slave-8vcz"
server_key: "1245a1286b46ed50663204e90dfdbd02"
server_key_iv: "cd0a02d5aabc159f64d2a2b6bf3b3d9c"
server_key_version: "encryptionkey@0"
I20260619 11:05:09.750808 19043 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:05:09.753239 19058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:09.754436 19043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:09.754565 19043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "eccbf7179e414a4a8cafc36f07ed9b18"
format_stamp: "Formatted at 2026-06-19 11:05:09 on dist-test-slave-8vcz"
server_key: "1245a1286b46ed50663204e90dfdbd02"
server_key_iv: "cd0a02d5aabc159f64d2a2b6bf3b3d9c"
server_key_version: "encryptionkey@0"
I20260619 11:05:09.754667 19043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:09.776468 19043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:09.779357 19043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:09.779558 19043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:09.786568 19043 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:35321
I20260619 11:05:09.786594 19110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:35321 every 8 connection(s)
I20260619 11:05:09.787500 19043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:09.790081 19111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:09.794762 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 19043
I20260619 11:05:09.794858 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:09.795125 16628 external_mini_cluster.cc:1428] Setting key 386f8b02416cc77a4c182ec327d79728
I20260619 11:05:09.795327 19111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18: Bootstrap starting.
I20260619 11:05:09.797761 19111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:09.798468 19111 log.cc:824] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:09.800702 19111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18: No bootstrap required, opened a new log
Jun 19 11:05:09 dist-test-slave-8vcz krb5kdc[19030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:09.805192 19111 raft_consensus.cc:348] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccbf7179e414a4a8cafc36f07ed9b18" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35321 } }
I20260619 11:05:09.805487 19111 raft_consensus.cc:374] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:09.805580 19111 raft_consensus.cc:729] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eccbf7179e414a4a8cafc36f07ed9b18, State: Initialized, Role: FOLLOWER
I20260619 11:05:09.806092 19111 consensus_queue.cc:260] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccbf7179e414a4a8cafc36f07ed9b18" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35321 } }
I20260619 11:05:09.806296 19111 raft_consensus.cc:388] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:09.806372 19111 raft_consensus.cc:482] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:09.806476 19111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:09.807639 19111 raft_consensus.cc:504] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccbf7179e414a4a8cafc36f07ed9b18" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35321 } }
I20260619 11:05:09.808009 19111 leader_election.cc:302] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eccbf7179e414a4a8cafc36f07ed9b18; no voters:
I20260619 11:05:09.808384 19111 leader_election.cc:288] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:09.808477 19116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:09.808727 19116 raft_consensus.cc:686] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [term 1 LEADER]: Becoming Leader. State: Replica: eccbf7179e414a4a8cafc36f07ed9b18, State: Running, Role: LEADER
I20260619 11:05:09.809056 19116 consensus_queue.cc:237] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccbf7179e414a4a8cafc36f07ed9b18" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35321 } }
I20260619 11:05:09.809680 19111 sys_catalog.cc:564] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:09.810956 19118 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eccbf7179e414a4a8cafc36f07ed9b18. Latest consensus state: current_term: 1 leader_uuid: "eccbf7179e414a4a8cafc36f07ed9b18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccbf7179e414a4a8cafc36f07ed9b18" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35321 } } }
I20260619 11:05:09.811015 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:09.796453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.16.61.62:35321)
0619 11:05:09.796870 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:09.796885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:09.796919 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:05:09.797526 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:09.797543 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:09.797853 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:09.798241 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:09.799105 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:09.800643 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:09.801560 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:09.802251 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:09.802425 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:09.805056 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:09.805062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:09.805077 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:09.805107 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:09.807167 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:09.807625 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:09.807631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:09.807636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:09.807808 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:09.808081 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:09.808084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:09.808086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:09.808610 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:09.809351 (+ 741us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:09.809676 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 11:05:09.809981 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260619 11:05:09.811111 19118 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:09.811465 19117 sys_catalog.cc:455] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eccbf7179e414a4a8cafc36f07ed9b18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eccbf7179e414a4a8cafc36f07ed9b18" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35321 } } }
I20260619 11:05:09.811583 19117 sys_catalog.cc:458] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18 [sys.catalog]: This master's current role is: LEADER
W20260619 11:05:09.815635 19131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:05:09.815773 19131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:05:09.815848 19127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:09.816527 19127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:09.821841 19127 catalog_manager.cc:1347] Generated new cluster ID: c9e3c6833fc84c91a93f077d763488c3
I20260619 11:05:09.821907 19127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:09.847704 19127 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:09.848971 19127 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:09.859570 19127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eccbf7179e414a4a8cafc36f07ed9b18: Generated new TSK 0
I20260619 11:05:09.860321 19127 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:09.898365 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35321
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:10.013312 19137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:10.013754 19137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:10.013854 19137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:10.019757 19137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:10.019912 19137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:10.020041 19137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:10.026073 19137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:10.027181 19137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:10.028331 19137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:10.039054 19145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:10.039042 19142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:10.039043 19143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:10.039646 19137 server_base.cc:1034] running on GCE node
I20260619 11:05:10.040097 19137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:10.040823 19137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:10.042052 19137 hybrid_clock.cc:648] HybridClock initialized: now 1781867110042023 us; error 48 us; skew 500 ppm
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:10.045540 19137 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:10.046589 19137 webserver.cc:460] Webserver started at http://127.16.61.1:35863/ using document root <none> and password file <none>
I20260619 11:05:10.047179 19137 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:10.047297 19137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:10.047511 19137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:10.049660 19137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "646cc47866b24c77b9427e787c6ad7b5"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "041046608d3ee63c61f41e5f3eda021f"
server_key_iv: "e2b99078ef3ed44695f3a943b01a9fa1"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.050253 19137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "646cc47866b24c77b9427e787c6ad7b5"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "041046608d3ee63c61f41e5f3eda021f"
server_key_iv: "e2b99078ef3ed44695f3a943b01a9fa1"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.054703 19137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 11:05:10.058383 19152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.060530 19137 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260619 11:05:10.060681 19137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "646cc47866b24c77b9427e787c6ad7b5"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "041046608d3ee63c61f41e5f3eda021f"
server_key_iv: "e2b99078ef3ed44695f3a943b01a9fa1"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.060866 19137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:10.084448 19137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:10.087898 19137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:10.088073 19137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:10.088665 19137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:10.089524 19137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:10.089579 19137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.089622 19137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:10.089639 19137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.099551 19137 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:36205
I20260619 11:05:10.099588 19265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:36205 every 8 connection(s)
I20260619 11:05:10.100548 19137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:10.107862 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 19137
I20260619 11:05:10.107973 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:10.108265 16628 external_mini_cluster.cc:1428] Setting key 2e3a6c4aa714cc164bde347514f02835
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:10.115775 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:10.102689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:51985 (local address 127.16.61.62:35321)
0619 11:05:10.102866 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:10.102870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:10.103742 (+ 872us) server_negotiation.cc:407] Connection header received
0619 11:05:10.105230 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:10.105234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:10.105314 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:10.105428 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:10.106586 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.107111 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.108488 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.108807 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.108897 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:10.111921 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:10.111927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:10.111931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:10.111964 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:10.113470 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:10.114147 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:10.114152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:10.114154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:10.114240 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:10.114648 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:10.114651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:10.114652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:10.114824 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:10.114898 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:10.115451 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 11:05:10.115586 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 11:05:10.116598 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:10.103052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35321 (local address 127.16.61.1:51985)
0619 11:05:10.103491 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:10.103530 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:10.104933 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:10.105602 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:10.105608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:10.105956 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:10.106426 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:10.106439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.107260 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.107265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:10.108354 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:10.108364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.108944 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.108948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:10.109118 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:10.109830 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:10.109859 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:10.111750 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:10.113590 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:10.113599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:10.113617 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:10.114006 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:10.114349 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:10.114354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:10.114359 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:10.114548 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:10.114927 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:10.114932 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:10.115195 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 11:05:10.115390 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 11:05:10.115727 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260619 11:05:10.117838 19266 heartbeater.cc:347] Connected to a master server at 127.16.61.62:35321
I20260619 11:05:10.118088 19266 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:05:10.118625 19266 heartbeater.cc:511] Master 127.16.61.62:35321 requested a full tablet report, sending...
I20260619 11:05:10.120339 19075 ts_manager.cc:194] Registered new tserver with Master: 646cc47866b24c77b9427e787c6ad7b5 (127.16.61.1:36205)
I20260619 11:05:10.121639 19075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:51985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:10.140177 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35321
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:10.250100 19271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:10.250355 19271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:10.250398 19271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:10.254410 19271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:10.254467 19271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:10.254523 19271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:10.259858 19271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:10.260823 19271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:10.261755 19271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:10.271420 19279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:10.271420 19277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:10.271461 19276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:10.271816 19271 server_base.cc:1034] running on GCE node
I20260619 11:05:10.272220 19271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:10.272848 19271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:10.274047 19271 hybrid_clock.cc:648] HybridClock initialized: now 1781867110274031 us; error 27 us; skew 500 ppm
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:10.276746 19271 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:10.277578 19271 webserver.cc:460] Webserver started at http://127.16.61.2:39495/ using document root <none> and password file <none>
I20260619 11:05:10.278134 19271 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:10.278230 19271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:10.278460 19271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:10.280517 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "b2552cdd3d2749028155dfaa63a218fc"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "b24f93289cfcd19d548f1dc1f1df664e"
server_key_iv: "3fa9ade775e2e03449c06506c407b140"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.281087 19271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "b2552cdd3d2749028155dfaa63a218fc"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "b24f93289cfcd19d548f1dc1f1df664e"
server_key_iv: "3fa9ade775e2e03449c06506c407b140"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.284806 19271 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:05:10.287163 19286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.288406 19271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:10.288540 19271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "b2552cdd3d2749028155dfaa63a218fc"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "b24f93289cfcd19d548f1dc1f1df664e"
server_key_iv: "3fa9ade775e2e03449c06506c407b140"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.288654 19271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:10.316268 19271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:10.319258 19271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:10.319484 19271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:10.320151 19271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:10.320978 19271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:10.321056 19271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.321121 19271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:10.321153 19271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.330789 19271 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:37927
I20260619 11:05:10.330838 19399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:37927 every 8 connection(s)
I20260619 11:05:10.331976 19271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:10.336683 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 19271
I20260619 11:05:10.336947 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:10.337411 16628 external_mini_cluster.cc:1428] Setting key 9865b902b6d6fbb77ea537ebdbf54c64
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
I20260619 11:05:10.347270 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:10.333887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:53141 (local address 127.16.61.62:35321)
0619 11:05:10.334015 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:10.334018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:10.335053 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 11:05:10.335921 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:10.335924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:10.336000 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:10.336107 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:10.338070 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.338605 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.339805 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.340038 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.340156 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:10.342706 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:10.342712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:10.342714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:10.342745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:10.344653 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:10.345300 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:10.345303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:10.345304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:10.345379 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:10.345864 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:10.345869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:10.345870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:10.346161 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:10.346292 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:10.347003 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 11:05:10.347130 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:05:10.348124 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:10.334284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35321 (local address 127.16.61.2:53141)
0619 11:05:10.334898 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:10.334937 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:10.335705 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:10.336390 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:10.336433 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:10.337249 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:10.337908 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:10.337924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.338743 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.338746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:10.339640 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:10.339649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.340193 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.340197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:10.340441 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:10.341101 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:10.341117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:10.342536 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:10.344795 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:10.344804 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:10.344822 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:10.345189 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:10.345482 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:10.345488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:10.345491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:10.345673 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:10.346329 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:10.346335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:10.346677 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 11:05:10.346910 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 11:05:10.347170 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:05:10.349520 19400 heartbeater.cc:347] Connected to a master server at 127.16.61.62:35321
I20260619 11:05:10.349774 19400 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:10.350253 19400 heartbeater.cc:511] Master 127.16.61.62:35321 requested a full tablet report, sending...
I20260619 11:05:10.351292 19075 ts_manager.cc:194] Registered new tserver with Master: b2552cdd3d2749028155dfaa63a218fc (127.16.61.2:37927)
I20260619 11:05:10.351945 19075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:53141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:05:10.366799 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35321
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:10.478042 19405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:10.478282 19405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:10.478324 19405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:10.482338 19405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:10.482395 19405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:10.482489 19405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:10.487636 19405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:10.488700 19405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:10.489698 19405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:10.499951 19413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:10.499955 19410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:10.499981 19411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:10.500173 19405 server_base.cc:1034] running on GCE node
I20260619 11:05:10.500645 19405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:10.501253 19405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:10.502460 19405 hybrid_clock.cc:648] HybridClock initialized: now 1781867110502443 us; error 29 us; skew 500 ppm
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:10.505187 19405 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:10.506023 19405 webserver.cc:460] Webserver started at http://127.16.61.3:33619/ using document root <none> and password file <none>
I20260619 11:05:10.506527 19405 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:10.506583 19405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:10.506735 19405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:10.508718 19405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "cd3618c96f194092aa771834f0a9c9ba"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "363bec07e7ce46e568056fcd325bb8ad"
server_key_iv: "63f9a97e14288dab8d1c999b02021cc5"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.509268 19405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "cd3618c96f194092aa771834f0a9c9ba"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "363bec07e7ce46e568056fcd325bb8ad"
server_key_iv: "63f9a97e14288dab8d1c999b02021cc5"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.513100 19405 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 11:05:10.515334 19420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.516569 19405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:10.516688 19405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "cd3618c96f194092aa771834f0a9c9ba"
format_stamp: "Formatted at 2026-06-19 11:05:10 on dist-test-slave-8vcz"
server_key: "363bec07e7ce46e568056fcd325bb8ad"
server_key_iv: "63f9a97e14288dab8d1c999b02021cc5"
server_key_version: "encryptionkey@0"
I20260619 11:05:10.516790 19405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:10.536913 19405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:10.540170 19405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:10.540369 19405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:10.541049 19405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:10.541847 19405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:10.541893 19405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.541934 19405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:10.541990 19405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:10.550760 19405 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:43999
I20260619 11:05:10.550791 19533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:43999 every 8 connection(s)
I20260619 11:05:10.551630 19405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:10.553443 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 19405
I20260619 11:05:10.553553 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:10.553855 16628 external_mini_cluster.cc:1428] Setting key 1c11c62dcde46ccf422f45e718719287
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867110, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:10.565472 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:10.553574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:60331 (local address 127.16.61.62:35321)
0619 11:05:10.553715 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:10.553719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:10.554673 (+ 954us) server_negotiation.cc:407] Connection header received
0619 11:05:10.556134 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:10.556137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:10.556189 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:10.556273 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:10.557393 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.557915 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.559077 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.559308 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.559426 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:10.561779 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:10.561782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:10.561784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:10.561805 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:10.563265 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:10.563817 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:10.563820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:10.563821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:10.563891 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:10.564285 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:10.564288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:10.564289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:10.564599 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:10.564686 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:10.565239 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 11:05:10.565334 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:05:10.566285 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:10.554018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35321 (local address 127.16.61.3:60331)
0619 11:05:10.554526 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:10.554578 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:10.555907 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:10.556434 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:10.556441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:10.556791 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:10.557244 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:10.557258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.558044 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.558048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:10.558941 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:10.558950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:10.559459 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:10.559463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:10.559631 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:10.560172 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:10.560188 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:10.561631 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:10.563382 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:10.563387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:10.563398 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:10.563645 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:10.564002 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:10.564004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:10.564006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:10.564154 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:10.564702 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:10.564707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:10.565010 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 11:05:10.565159 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 11:05:10.565402 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":158,"threads_started":1}
I20260619 11:05:10.567428 19534 heartbeater.cc:347] Connected to a master server at 127.16.61.62:35321
I20260619 11:05:10.567700 19534 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:10.568163 19534 heartbeater.cc:511] Master 127.16.61.62:35321 requested a full tablet report, sending...
I20260619 11:05:10.569164 19075 ts_manager.cc:194] Registered new tserver with Master: cd3618c96f194092aa771834f0a9c9ba (127.16.61.3:43999)
I20260619 11:05:10.569739 19075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:60331
I20260619 11:05:10.579608 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:05:10.592994 19114 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:10.591504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.16.61.62:35321)
0619 11:05:10.591710 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:10.591716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:10.591791 (+ 75us) server_negotiation.cc:407] Connection header received
0619 11:05:10.591923 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:10.591926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:10.591969 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:10.592048 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:10.592871 (+ 823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48912: BlockingRecv error: recv got EOF from 127.0.0.1:48912 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 11:05:10.594100 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 19137
I20260619 11:05:10.599334 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 19271
I20260619 11:05:10.604602 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 19405
I20260619 11:05:10.609543 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 19043
2026-06-19T11:05:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2162 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19546](info): set up 2 sockets
Jun 19 11:05:10 dist-test-slave-8vcz krb5kdc[19546](info): commencing operation
krb5kdc: starting...
W20260619 11:05:11.659883 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:11 dist-test-slave-8vcz krb5kdc[19546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:11.778244 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:42637
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:42637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:11.884857 19559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:11.885165 19559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:11.885239 19559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:11.889321 19559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:11.889406 19559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:11.889444 19559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:11.889503 19559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:11.889544 19559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:11.894688 19559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:42637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:11.895803 19559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:11.896802 19559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:11.904857 19564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:11.904987 19559 server_base.cc:1034] running on GCE node
W20260619 11:05:11.904852 19567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:11.904852 19565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:11.905658 19559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:11.906633 19559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:11.907739 19559 hybrid_clock.cc:648] HybridClock initialized: now 1781867111907705 us; error 46 us; skew 500 ppm
Jun 19 11:05:11 dist-test-slave-8vcz krb5kdc[19546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867111, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:11.913959 19559 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:11.914891 19559 webserver.cc:460] Webserver started at http://127.16.61.62:33877/ using document root <none> and password file <none>
I20260619 11:05:11.915459 19559 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:11.915551 19559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:11.915827 19559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:11.917843 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c"
format_stamp: "Formatted at 2026-06-19 11:05:11 on dist-test-slave-8vcz"
server_key: "9c9ae1b9f83ad15a578ba68e82742d0a"
server_key_iv: "83b014d5aaba9d2705d29f313724f5d9"
server_key_version: "encryptionkey@0"
I20260619 11:05:11.918421 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c"
format_stamp: "Formatted at 2026-06-19 11:05:11 on dist-test-slave-8vcz"
server_key: "9c9ae1b9f83ad15a578ba68e82742d0a"
server_key_iv: "83b014d5aaba9d2705d29f313724f5d9"
server_key_version: "encryptionkey@0"
I20260619 11:05:11.922062 19559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:11.924515 19574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:11.925690 19559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:05:11.925825 19559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c"
format_stamp: "Formatted at 2026-06-19 11:05:11 on dist-test-slave-8vcz"
server_key: "9c9ae1b9f83ad15a578ba68e82742d0a"
server_key_iv: "83b014d5aaba9d2705d29f313724f5d9"
server_key_version: "encryptionkey@0"
I20260619 11:05:11.925944 19559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:11.943903 19559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:11.946673 19559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:11.946866 19559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:11.953903 19559 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:42637
I20260619 11:05:11.954181 19626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:42637 every 8 connection(s)
I20260619 11:05:11.954922 19559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:11.957683 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:11.962862 19627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c: Bootstrap starting.
I20260619 11:05:11.964025 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 19559
I20260619 11:05:11.964113 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:11.964390 16628 external_mini_cluster.cc:1428] Setting key b6b0cb93d210fb707da18ca4a85e0720
I20260619 11:05:11.965569 19627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:11.966264 19627 log.cc:824] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:11.968452 19627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c: No bootstrap required, opened a new log
I20260619 11:05:11.972761 19627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42637 } }
I20260619 11:05:11.973021 19627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:11.973104 19627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba0d4b3b7b94c8c90bd93949ca52a5c, State: Initialized, Role: FOLLOWER
Jun 19 11:05:11 dist-test-slave-8vcz krb5kdc[19546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:11.973582 19627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42637 } }
I20260619 11:05:11.973739 19627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:11.973827 19627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:11.973941 19627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:11.975083 19627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42637 } }
I20260619 11:05:11.975425 19627 leader_election.cc:302] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ba0d4b3b7b94c8c90bd93949ca52a5c; no voters:
I20260619 11:05:11.975786 19627 leader_election.cc:288] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:11.975862 19632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:11.976069 19632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [term 1 LEADER]: Becoming Leader. State: Replica: 1ba0d4b3b7b94c8c90bd93949ca52a5c, State: Running, Role: LEADER
I20260619 11:05:11.976352 19632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42637 } }
I20260619 11:05:11.977026 19627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:11.977512 19632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ba0d4b3b7b94c8c90bd93949ca52a5c. Latest consensus state: current_term: 1 leader_uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42637 } } }
I20260619 11:05:11.977625 19632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:11.978018 19636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:11.978516 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ba0d4b3b7b94c8c90bd93949ca52a5c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42637 } } }
I20260619 11:05:11.978689 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:11.979308 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:11.965441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.16.61.62:42637)
0619 11:05:11.965901 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:11.965916 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:11.965950 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:05:11.966586 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:11.966629 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:11.967033 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:11.967328 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:11.968241 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:11.969261 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:11.970470 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:11.970922 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:11.971041 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:11.973946 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:11.973951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:11.973963 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:11.973989 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:11.975898 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:11.976420 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:11.976427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:11.976433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:11.976565 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:11.976852 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:11.976854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:11.976855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:11.977290 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:11.977502 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:11.977878 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 11:05:11.978251 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":145,"threads_started":1}
I20260619 11:05:11.981154 19636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:11.982646 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:05:11.986130 19636 catalog_manager.cc:1347] Generated new cluster ID: 5fa0636166914d8ba01f5fc76f4744c2
I20260619 11:05:11.986186 19636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:12.001792 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:11.996272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.16.61.62:42637)
0619 11:05:11.996383 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:11.996387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:11.996648 (+ 261us) server_negotiation.cc:407] Connection header received
0619 11:05:11.996745 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:11.996749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:11.996806 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:11.996895 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:11.997434 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:11.998259 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:11.999371 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:11.999752 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:11.999856 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:12.000750 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:12.000754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:12.000757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:12.001346 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:12.001379 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:12.001397 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:12.001487 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:12.001576 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 11:05:12.001637 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":39}
I20260619 11:05:12.005818 19636 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:12.006729 19636 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:12.010761 19636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ba0d4b3b7b94c8c90bd93949ca52a5c: Generated new TSK 0
I20260619 11:05:12.011432 19636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:05:12.019521 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 19559
2026-06-19T11:05:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1407 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:12 dist-test-slave-8vcz krb5kdc[19660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:12 dist-test-slave-8vcz krb5kdc[19660](info): set up 2 sockets
Jun 19 11:05:12 dist-test-slave-8vcz krb5kdc[19660](info): commencing operation
krb5kdc: starting...
W20260619 11:05:13.055805 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:13 dist-test-slave-8vcz krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:13.172828 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:43551
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:43551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:13.277890 19673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:13.278183 19673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:13.278256 19673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:13.282287 19673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:13.282377 19673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:13.282415 19673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:13.282474 19673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:13.282516 19673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:13.287606 19673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:43551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:43551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:13.288648 19673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:13.289634 19673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:13.297917 19678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:13.297912 19681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:13.297910 19679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:13.298204 19673 server_base.cc:1034] running on GCE node
I20260619 11:05:13.298777 19673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:13.299916 19673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:13.301111 19673 hybrid_clock.cc:648] HybridClock initialized: now 1781867113301101 us; error 28 us; skew 500 ppm
Jun 19 11:05:13 dist-test-slave-8vcz krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867113, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:13.303706 19673 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:13.304543 19673 webserver.cc:460] Webserver started at http://127.16.61.62:44481/ using document root <none> and password file <none>
I20260619 11:05:13.305088 19673 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:13.305182 19673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:13.305406 19673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:13.307400 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "b6fa31891c1042adb8186331035cc3fe"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "3ff3966a38f094db45b815335d7e1d42"
server_key_iv: "35b24c04d80c59e4829819fba3ff3de1"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.307982 19673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "b6fa31891c1042adb8186331035cc3fe"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "3ff3966a38f094db45b815335d7e1d42"
server_key_iv: "35b24c04d80c59e4829819fba3ff3de1"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.311384 19673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:13.313689 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:13.314800 19673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:13.314951 19673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "b6fa31891c1042adb8186331035cc3fe"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "3ff3966a38f094db45b815335d7e1d42"
server_key_iv: "35b24c04d80c59e4829819fba3ff3de1"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.315065 19673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:13.331363 19673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:13.334281 19673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:13.334488 19673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:13.341907 19673 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:43551
I20260619 11:05:13.342211 19740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:43551 every 8 connection(s)
I20260619 11:05:13.343281 19673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:13.345906 19741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:13.348066 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 19673
I20260619 11:05:13.348179 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:13.348452 16628 external_mini_cluster.cc:1428] Setting key 15d9bc4012dabef16f923f1977543768
I20260619 11:05:13.351935 19741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe: Bootstrap starting.
I20260619 11:05:13.353967 19741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:13.354657 19741 log.cc:824] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:13.356472 19741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe: No bootstrap required, opened a new log
Jun 19 11:05:13 dist-test-slave-8vcz krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:13.359983 19741 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fa31891c1042adb8186331035cc3fe" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43551 } }
I20260619 11:05:13.360262 19741 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:13.360324 19741 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fa31891c1042adb8186331035cc3fe, State: Initialized, Role: FOLLOWER
I20260619 11:05:13.360757 19741 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fa31891c1042adb8186331035cc3fe" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43551 } }
I20260619 11:05:13.360939 19741 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:13.361009 19741 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:13.361150 19741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:13.362182 19741 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fa31891c1042adb8186331035cc3fe" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43551 } }
I20260619 11:05:13.362498 19741 leader_election.cc:302] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6fa31891c1042adb8186331035cc3fe; no voters:
I20260619 11:05:13.362768 19741 leader_election.cc:288] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:13.362874 19746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:13.363076 19746 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [term 1 LEADER]: Becoming Leader. State: Replica: b6fa31891c1042adb8186331035cc3fe, State: Running, Role: LEADER
I20260619 11:05:13.363096 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:13.349753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.16.61.62:43551)
0619 11:05:13.350156 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:13.350171 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:13.350203 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:05:13.350840 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:13.350864 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:13.351308 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:13.351658 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:13.352529 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.353501 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.354406 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.354929 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.355100 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:13.357724 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:13.357729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:13.357741 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:13.357766 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:13.359986 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:13.360479 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:13.360483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:13.360487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:13.360642 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:13.360956 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:13.360959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:13.360960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:13.361434 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:13.361582 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:13.361871 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 11:05:13.362169 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260619 11:05:13.363564 19746 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fa31891c1042adb8186331035cc3fe" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43551 } }
I20260619 11:05:13.364040 19741 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:13.365795 19748 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fa31891c1042adb8186331035cc3fe. Latest consensus state: current_term: 1 leader_uuid: "b6fa31891c1042adb8186331035cc3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fa31891c1042adb8186331035cc3fe" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43551 } } }
I20260619 11:05:13.365913 19748 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:13.366370 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6fa31891c1042adb8186331035cc3fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fa31891c1042adb8186331035cc3fe" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43551 } } }
I20260619 11:05:13.366652 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe [sys.catalog]: This master's current role is: LEADER
W20260619 11:05:13.368739 19760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:05:13.368834 19760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:05:13.368909 19762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:13.369590 19762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:13.374678 19762 catalog_manager.cc:1347] Generated new cluster ID: 21af9103b9b1430f9562f8651cc84a57
I20260619 11:05:13.374738 19762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:13.384353 19762 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:13.385469 19762 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:13.395923 19762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6fa31891c1042adb8186331035cc3fe: Generated new TSK 0
I20260619 11:05:13.396639 19762 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:13.426221 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43551
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:13.532856 19767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:13.533131 19767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:13.533210 19767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:13.537199 19767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:13.537254 19767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:13.537312 19767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:13.542479 19767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:13.543413 19767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:13.544341 19767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:13.553787 19772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:13.553814 19773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:13.553799 19775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:13.554119 19767 server_base.cc:1034] running on GCE node
I20260619 11:05:13.554592 19767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:13.555227 19767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:13.556420 19767 hybrid_clock.cc:648] HybridClock initialized: now 1781867113556404 us; error 27 us; skew 500 ppm
Jun 19 11:05:13 dist-test-slave-8vcz krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867113, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:13.559129 19767 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:13.560092 19767 webserver.cc:460] Webserver started at http://127.16.61.1:38519/ using document root <none> and password file <none>
I20260619 11:05:13.560650 19767 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:13.560743 19767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:13.560972 19767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:13.562943 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "f140e55a7d6b4ce596b39dc696669a44"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "244b7a5915b8e361b357cdb857b4e416"
server_key_iv: "9f5f66bd19ca768f008e42b13e4ce8fb"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.563504 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "f140e55a7d6b4ce596b39dc696669a44"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "244b7a5915b8e361b357cdb857b4e416"
server_key_iv: "9f5f66bd19ca768f008e42b13e4ce8fb"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.567479 19767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 11:05:13.569923 19782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:13.571086 19767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:13.571233 19767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "f140e55a7d6b4ce596b39dc696669a44"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "244b7a5915b8e361b357cdb857b4e416"
server_key_iv: "9f5f66bd19ca768f008e42b13e4ce8fb"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.571348 19767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:13.591833 19767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:13.594574 19767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:13.594774 19767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:13.595351 19767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:13.596210 19767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:13.596284 19767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:13.596354 19767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:13.596405 19767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:13.605394 19767 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:35073
I20260619 11:05:13.605418 19895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:35073 every 8 connection(s)
I20260619 11:05:13.606247 19767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:13.612334 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 19767
I20260619 11:05:13.612459 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:13.612725 16628 external_mini_cluster.cc:1428] Setting key 0e6150733f92c94b997de7927d9ece3c
Jun 19 11:05:13 dist-test-slave-8vcz krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867113, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:13.620177 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:13.608051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:57541 (local address 127.16.61.62:43551)
0619 11:05:13.608188 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:13.608197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:13.609015 (+ 818us) server_negotiation.cc:407] Connection header received
0619 11:05:13.610343 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:13.610347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:13.610402 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:13.610541 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:13.611661 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.612218 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.613500 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.613732 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.613848 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:13.616400 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:13.616405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:13.616409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:13.616453 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:13.617962 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:13.618575 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:13.618581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:13.618584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:13.618665 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:13.619018 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:13.619021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:13.619023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:13.619197 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:13.619328 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:13.619892 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 11:05:13.620012 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:05:13.620990 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:13.608420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43551 (local address 127.16.61.1:57541)
0619 11:05:13.608862 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:13.608903 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:13.610132 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:13.610710 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:13.610717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:13.611061 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:13.611512 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:13.611525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.612347 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.612350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:13.613364 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:13.613373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.613881 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.613885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:13.614055 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:13.614647 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:13.614663 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:13.616235 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:13.618115 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:13.618121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:13.618135 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:13.618489 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:13.618759 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:13.618764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:13.618766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:13.618905 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:13.619303 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:13.619310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:13.619585 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 11:05:13.619841 (+ 256us) client_negotiation.cc:240] Negotiation successful
0619 11:05:13.620153 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:05:13.622313 19896 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43551
I20260619 11:05:13.622543 19896 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:13.623000 19896 heartbeater.cc:511] Master 127.16.61.62:43551 requested a full tablet report, sending...
I20260619 11:05:13.624468 19705 ts_manager.cc:194] Registered new tserver with Master: f140e55a7d6b4ce596b39dc696669a44 (127.16.61.1:35073)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
I20260619 11:05:13.625612 19705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:57541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:13.638707 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43551
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:13.745893 19901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:13.746191 19901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:13.746268 19901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:13.750315 19901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:13.750406 19901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:13.750496 19901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:13.755626 19901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:13.756726 19901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:13.757727 19901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:13.767202 19906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:13.767230 19907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:13.767230 19909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:13.767504 19901 server_base.cc:1034] running on GCE node
I20260619 11:05:13.767923 19901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:13.768535 19901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:13.769727 19901 hybrid_clock.cc:648] HybridClock initialized: now 1781867113769712 us; error 26 us; skew 500 ppm
Jun 19 11:05:13 dist-test-slave-8vcz krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867113, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:13.772445 19901 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:13.773299 19901 webserver.cc:460] Webserver started at http://127.16.61.2:41791/ using document root <none> and password file <none>
I20260619 11:05:13.773878 19901 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:13.773979 19901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:13.774202 19901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:13.776249 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "c5bf7b3962c74d24a21b5b99fb3ce84a"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "0a19a26076443d0646ea82618ada1dfe"
server_key_iv: "939f14584206a06d6c9f656c68b61caf"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.776803 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "c5bf7b3962c74d24a21b5b99fb3ce84a"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "0a19a26076443d0646ea82618ada1dfe"
server_key_iv: "939f14584206a06d6c9f656c68b61caf"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.780375 19901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:13.782713 19916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:13.783891 19901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:13.784016 19901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "c5bf7b3962c74d24a21b5b99fb3ce84a"
format_stamp: "Formatted at 2026-06-19 11:05:13 on dist-test-slave-8vcz"
server_key: "0a19a26076443d0646ea82618ada1dfe"
server_key_iv: "939f14584206a06d6c9f656c68b61caf"
server_key_version: "encryptionkey@0"
I20260619 11:05:13.784139 19901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:13.814342 19901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:13.817345 19901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:13.817553 19901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:13.818192 19901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:13.819178 19901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:13.819227 19901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:13.819303 19901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:13.819339 19901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:13.828059 19901 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:44851
I20260619 11:05:13.828115 20029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:44851 every 8 connection(s)
I20260619 11:05:13.828964 19901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:13.834863 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 19901
I20260619 11:05:13.834981 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:13.835268 16628 external_mini_cluster.cc:1428] Setting key 2033884a5c6e172c6cc0a84ba0f037d4
Jun 19 11:05:13 dist-test-slave-8vcz krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867113, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:13.843391 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:13.830849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:58861 (local address 127.16.61.62:43551)
0619 11:05:13.831015 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:13.831019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:13.831875 (+ 856us) server_negotiation.cc:407] Connection header received
0619 11:05:13.833674 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:13.833677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:13.833721 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:13.833797 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:13.834875 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.835633 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.837105 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.837355 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.837474 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:13.839911 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:13.839915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:13.839917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:13.839942 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:13.841405 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:13.841923 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:13.841927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:13.841929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:13.842009 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:13.842358 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:13.842363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:13.842365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:13.842542 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:13.842653 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:13.843123 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 11:05:13.843236 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
I20260619 11:05:13.844218 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:13.831240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43551 (local address 127.16.61.2:58861)
0619 11:05:13.831691 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:13.831769 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:13.833428 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:13.833945 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:13.833951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:13.834286 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:13.834727 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:13.834739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.835780 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.835784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:13.836976 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:13.836986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:13.837506 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:13.837510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:13.837686 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:13.838284 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:13.838300 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:13.839752 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:13.841544 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:13.841550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:13.841563 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:13.841815 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:13.842113 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:13.842116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:13.842118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:13.842252 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:13.842651 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:13.842658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:13.842887 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 11:05:13.843079 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 11:05:13.843386 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260619 11:05:13.845299 20030 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43551
I20260619 11:05:13.845556 20030 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:13.846026 20030 heartbeater.cc:511] Master 127.16.61.62:43551 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:05:13.847112 19705 ts_manager.cc:194] Registered new tserver with Master: c5bf7b3962c74d24a21b5b99fb3ce84a (127.16.61.2:44851)
I20260619 11:05:13.847976 19705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:58861
I20260619 11:05:14.627779 19896 heartbeater.cc:503] Master 127.16.61.62:43551 was elected leader, sending a full tablet report...
I20260619 11:05:14.850042 20030 heartbeater.cc:503] Master 127.16.61.62:43551 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
W20260619 11:05:14.861421 16628 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.61.3: real 1.024s user 0.000s sys 0.003s
I20260619 11:05:14.861948 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43551
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:14.968531 20035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:14.968760 20035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:14.968807 20035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:14.972687 20035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:14.972740 20035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:14.972802 20035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:14.978093 20035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:14.979014 20035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:14.979987 20035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:14.989171 20041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:14.989164 20043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:14.989409 20035 server_base.cc:1034] running on GCE node
W20260619 11:05:14.989338 20040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:14.989791 20035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:14.990391 20035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:14.991537 20035 hybrid_clock.cc:648] HybridClock initialized: now 1781867114991519 us; error 29 us; skew 500 ppm
Jun 19 11:05:14 dist-test-slave-8vcz krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867114, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:14.994292 20035 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:14.995090 20035 webserver.cc:460] Webserver started at http://127.16.61.3:35117/ using document root <none> and password file <none>
I20260619 11:05:14.995579 20035 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:14.995636 20035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:14.995865 20035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:14.997841 20035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "5c081b30078d4bcd99a97879e12c42dc"
format_stamp: "Formatted at 2026-06-19 11:05:14 on dist-test-slave-8vcz"
server_key: "2c0ff11dda5470f05c079ab1ee8fb142"
server_key_iv: "42915c50295d150e49c2a4334484a87b"
server_key_version: "encryptionkey@0"
I20260619 11:05:14.998392 20035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "5c081b30078d4bcd99a97879e12c42dc"
format_stamp: "Formatted at 2026-06-19 11:05:14 on dist-test-slave-8vcz"
server_key: "2c0ff11dda5470f05c079ab1ee8fb142"
server_key_iv: "42915c50295d150e49c2a4334484a87b"
server_key_version: "encryptionkey@0"
I20260619 11:05:15.002365 20035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 11:05:15.004843 20050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:15.006104 20035 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:15.006240 20035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "5c081b30078d4bcd99a97879e12c42dc"
format_stamp: "Formatted at 2026-06-19 11:05:14 on dist-test-slave-8vcz"
server_key: "2c0ff11dda5470f05c079ab1ee8fb142"
server_key_iv: "42915c50295d150e49c2a4334484a87b"
server_key_version: "encryptionkey@0"
I20260619 11:05:15.006369 20035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:15.019521 20035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:15.022806 20035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:15.023012 20035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:15.023582 20035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:15.024374 20035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:15.024453 20035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:15.024519 20035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:15.024570 20035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:15.032799 20035 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:42995
I20260619 11:05:15.032866 20163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:42995 every 8 connection(s)
I20260619 11:05:15.033679 20035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:15.037988 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 20035
I20260619 11:05:15.038120 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:15.038408 16628 external_mini_cluster.cc:1428] Setting key 0625db37f07e5ada762db09bc4a59b68
Jun 19 11:05:15 dist-test-slave-8vcz krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867114, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:15.047065 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:15.035359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:57301 (local address 127.16.61.62:43551)
0619 11:05:15.035596 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:15.035600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:15.036392 (+ 792us) server_negotiation.cc:407] Connection header received
0619 11:05:15.037729 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:15.037736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:15.037798 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:15.037906 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:15.039262 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:15.040036 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:15.041264 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:15.041483 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:15.041575 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:15.043841 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:15.043844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:15.043847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:15.043872 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:15.045125 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:15.045617 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:15.045622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:15.045624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:15.045704 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:15.046030 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:15.046033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:15.046035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:15.046206 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:15.046309 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:15.046764 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 11:05:15.046879 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":58,"threads_started":1}
I20260619 11:05:15.047843 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:15.035807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43551 (local address 127.16.61.3:57301)
0619 11:05:15.036241 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:15.036288 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:15.037518 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:15.038145 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:15.038171 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:15.038585 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:15.039136 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:15.039149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:15.040137 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:15.040141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:15.041151 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:15.041161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:15.041618 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:15.041621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:15.041751 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:15.042247 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:15.042262 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:15.043639 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:15.045256 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:15.045262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:15.045274 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:15.045516 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:15.045804 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:15.045806 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:15.045808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:15.045932 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:15.046317 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:15.046326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:15.046544 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 11:05:15.046719 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 11:05:15.047012 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":119,"threads_started":1}
I20260619 11:05:15.048885 20164 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43551
I20260619 11:05:15.049073 20164 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:15.049479 20164 heartbeater.cc:511] Master 127.16.61.62:43551 requested a full tablet report, sending...
I20260619 11:05:15.050361 19705 ts_manager.cc:194] Registered new tserver with Master: 5c081b30078d4bcd99a97879e12c42dc (127.16.61.3:42995)
I20260619 11:05:15.050953 19705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:57301
I20260619 11:05:15.052412 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:05:15 dist-test-slave-8vcz krb5kdc[19660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867115, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
W20260619 11:05:15.158991 16628 mini_kdc.cc:317] Time spent kinit for joe-interloper: real 0.106s user 0.002s sys 0.002s
Jun 19 11:05:15 dist-test-slave-8vcz krb5kdc[19660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867115, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:15.169874 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:15.160982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.16.61.62:43551)
0619 11:05:15.161165 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:15.161170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:15.161374 (+ 204us) server_negotiation.cc:407] Connection header received
0619 11:05:15.161487 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:15.161490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:15.161535 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:15.161609 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:15.162533 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:15.163067 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:15.163968 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:15.164202 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:15.164281 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:15.166879 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:15.166882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:15.166884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:15.166907 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:15.168303 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:15.168681 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:15.168684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:15.168685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:15.168756 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:15.169047 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:15.169049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:15.169050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:15.169292 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:15.169402 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:15.169578 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 11:05:15.169673 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 11:05:15.170288 19704 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44274
I20260619 11:05:15.171201 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 19767
I20260619 11:05:15.176949 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 19901
I20260619 11:05:15.182210 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 20035
I20260619 11:05:15.187183 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 19673
2026-06-19T11:05:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3169 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:15 dist-test-slave-8vcz krb5kdc[20177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:15 dist-test-slave-8vcz krb5kdc[20177](info): set up 2 sockets
Jun 19 11:05:15 dist-test-slave-8vcz krb5kdc[20177](info): commencing operation
krb5kdc: starting...
W20260619 11:05:16.695832 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.486s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:16 dist-test-slave-8vcz krb5kdc[20177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:16.812736 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:42603
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:42603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:16.915725 20190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:16.915999 20190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:16.916070 20190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:16.920131 20190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:16.920270 20190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:16.920322 20190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:16.920354 20190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:16.920389 20190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:16.925463 20190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:42603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:42603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:16.926571 20190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:16.927548 20190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:16.935544 20195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:16.935557 20196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:16.935745 20190 server_base.cc:1034] running on GCE node
W20260619 11:05:16.935974 20198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:16.936455 20190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:16.937551 20190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:16.938738 20190 hybrid_clock.cc:648] HybridClock initialized: now 1781867116938725 us; error 24 us; skew 500 ppm
Jun 19 11:05:16 dist-test-slave-8vcz krb5kdc[20177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867116, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:16.945019 20190 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:16.945858 20190 webserver.cc:460] Webserver started at http://127.16.61.62:34331/ using document root <none> and password file <none>
I20260619 11:05:16.946424 20190 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:16.946514 20190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:16.946738 20190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:16.948815 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "7493aa7fd94844d0ad874992a38253ba"
format_stamp: "Formatted at 2026-06-19 11:05:16 on dist-test-slave-8vcz"
server_key: "8eb43bba7ef5145ffc4b239f4bd56019"
server_key_iv: "6671d109398d09650a422d7e0e76c32b"
server_key_version: "encryptionkey@0"
I20260619 11:05:16.949373 20190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "7493aa7fd94844d0ad874992a38253ba"
format_stamp: "Formatted at 2026-06-19 11:05:16 on dist-test-slave-8vcz"
server_key: "8eb43bba7ef5145ffc4b239f4bd56019"
server_key_iv: "6671d109398d09650a422d7e0e76c32b"
server_key_version: "encryptionkey@0"
I20260619 11:05:16.952973 20190 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:05:16.955252 20205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:16.956549 20190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:05:16.956682 20190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "7493aa7fd94844d0ad874992a38253ba"
format_stamp: "Formatted at 2026-06-19 11:05:16 on dist-test-slave-8vcz"
server_key: "8eb43bba7ef5145ffc4b239f4bd56019"
server_key_iv: "6671d109398d09650a422d7e0e76c32b"
server_key_version: "encryptionkey@0"
I20260619 11:05:16.956787 20190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:16.973611 20190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:16.976660 20190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:16.976862 20190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:16.983881 20190 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:42603
I20260619 11:05:16.983904 20257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:42603 every 8 connection(s)
I20260619 11:05:16.984787 20190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:16.987345 20258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:16.987859 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 20190
I20260619 11:05:16.987954 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:16.988256 16628 external_mini_cluster.cc:1428] Setting key a49e119054df3e75d66109b561ff4a33
I20260619 11:05:16.992955 20258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba: Bootstrap starting.
I20260619 11:05:16.994985 20258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:16.995800 20258 log.cc:824] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:05:16 dist-test-slave-8vcz krb5kdc[20177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:16.998011 20258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba: No bootstrap required, opened a new log
I20260619 11:05:17.001972 20258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7493aa7fd94844d0ad874992a38253ba" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42603 } }
I20260619 11:05:17.002148 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:16.989578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.16.61.62:42603)
0619 11:05:16.989993 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:16.990006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:16.990041 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:05:16.990702 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:16.990721 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:16.991026 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:16.991393 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:16.992344 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:16.993359 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:16.994177 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:16.994629 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:16.994805 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:16.997698 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:16.997703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:16.997715 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:16.997738 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:16.999328 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:16.999881 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:16.999886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:16.999890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:16.999995 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.000301 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.000303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.000304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.000660 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:17.000803 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:17.001079 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 11:05:17.001472 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260619 11:05:17.002172 20258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:17.002305 20258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7493aa7fd94844d0ad874992a38253ba, State: Initialized, Role: FOLLOWER
I20260619 11:05:17.002701 20258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7493aa7fd94844d0ad874992a38253ba" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42603 } }
I20260619 11:05:17.002868 20258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:17.002954 20258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:17.003073 20258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:17.004146 20258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7493aa7fd94844d0ad874992a38253ba" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42603 } }
I20260619 11:05:17.004487 20258 leader_election.cc:302] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7493aa7fd94844d0ad874992a38253ba; no voters:
I20260619 11:05:17.004796 20258 leader_election.cc:288] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:17.004870 20263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:17.005100 20263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [term 1 LEADER]: Becoming Leader. State: Replica: 7493aa7fd94844d0ad874992a38253ba, State: Running, Role: LEADER
I20260619 11:05:17.005426 20263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7493aa7fd94844d0ad874992a38253ba" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42603 } }
I20260619 11:05:17.005944 20258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:17.006971 20265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7493aa7fd94844d0ad874992a38253ba. Latest consensus state: current_term: 1 leader_uuid: "7493aa7fd94844d0ad874992a38253ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7493aa7fd94844d0ad874992a38253ba" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42603 } } }
I20260619 11:05:17.007091 20265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:17.007468 20264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7493aa7fd94844d0ad874992a38253ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7493aa7fd94844d0ad874992a38253ba" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42603 } } }
I20260619 11:05:17.007603 20264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:17.008057 20272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:17.010984 20272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:17.015928 20272 catalog_manager.cc:1347] Generated new cluster ID: 1d36e3d063ec445eb17b815bf1d3664e
I20260619 11:05:17.015990 20272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:17.051651 20272 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:17.052848 20272 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:17.060484 20272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7493aa7fd94844d0ad874992a38253ba: Generated new TSK 0
I20260619 11:05:17.061195 20272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:17.089876 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42603
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:17.219882 20284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:17.220113 20284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:17.220152 20284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:17.224017 20284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:17.224063 20284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:17.224118 20284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:17.229806 20284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:17.230680 20284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:17.231562 20284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:17.240470 20289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:17.240553 20290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:17.240543 20292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:17.240818 20284 server_base.cc:1034] running on GCE node
I20260619 11:05:17.241158 20284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:17.241706 20284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:17.242871 20284 hybrid_clock.cc:648] HybridClock initialized: now 1781867117242858 us; error 26 us; skew 500 ppm
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:17.245591 20284 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:17.246460 20284 webserver.cc:460] Webserver started at http://127.16.61.1:45551/ using document root <none> and password file <none>
I20260619 11:05:17.246986 20284 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:17.247049 20284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:17.247362 20284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:17.249387 20284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "22b9e6ac45d0402d9be3bbe9b2fc671f"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "6209724fa1e3aa0a8c14112a84ebaa64"
server_key_iv: "c4a7000a3b6ee421a92dc32d685b1d26"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.249940 20284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "22b9e6ac45d0402d9be3bbe9b2fc671f"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "6209724fa1e3aa0a8c14112a84ebaa64"
server_key_iv: "c4a7000a3b6ee421a92dc32d685b1d26"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.253839 20284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:05:17.256166 20299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.257503 20284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:17.257650 20284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "22b9e6ac45d0402d9be3bbe9b2fc671f"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "6209724fa1e3aa0a8c14112a84ebaa64"
server_key_iv: "c4a7000a3b6ee421a92dc32d685b1d26"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.257761 20284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:17.289170 20284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:17.292438 20284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:17.292636 20284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:17.293200 20284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:17.293983 20284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:17.294052 20284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.294121 20284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:17.294174 20284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.303042 20284 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:34751
I20260619 11:05:17.303069 20412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:34751 every 8 connection(s)
I20260619 11:05:17.303951 20284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:17.308305 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 20284
I20260619 11:05:17.308429 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:17.308730 16628 external_mini_cluster.cc:1428] Setting key 482358658bc98020a63e3b00aec1804e
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:17.318557 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:17.305557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:45013 (local address 127.16.61.62:42603)
0619 11:05:17.305692 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:17.305696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:17.306598 (+ 902us) server_negotiation.cc:407] Connection header received
0619 11:05:17.308051 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:17.308054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:17.308122 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:17.308240 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:17.309648 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.310322 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.312054 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.312274 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.312382 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:17.314849 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:17.314853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:17.314855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:17.314882 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:17.316408 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.316985 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.316988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.316989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.317061 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.317468 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.317471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.317472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.317636 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:17.317713 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:17.318331 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 11:05:17.318442 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:05:17.319422 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:17.305957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42603 (local address 127.16.61.1:45013)
0619 11:05:17.306428 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:17.306467 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:17.307873 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:17.308446 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:17.308455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:17.308918 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:17.309517 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:17.309531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.310464 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.310468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:17.311929 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:17.311938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.312420 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.312424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:17.312580 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:17.313067 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:17.313091 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:17.314648 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:17.316528 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:17.316533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:17.316547 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:17.316884 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:17.317158 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:17.317163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:17.317167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:17.317363 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:17.317748 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:17.317757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:17.318065 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 11:05:17.318287 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 11:05:17.318606 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260619 11:05:17.320647 20413 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42603
I20260619 11:05:17.320868 20413 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
I20260619 11:05:17.321346 20413 heartbeater.cc:511] Master 127.16.61.62:42603 requested a full tablet report, sending...
I20260619 11:05:17.322829 20222 ts_manager.cc:194] Registered new tserver with Master: 22b9e6ac45d0402d9be3bbe9b2fc671f (127.16.61.1:34751)
I20260619 11:05:17.323962 20222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:45013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:17.336151 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42603
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:17.441057 20418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:17.441309 20418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:17.441375 20418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:17.445309 20418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:17.445381 20418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:17.445477 20418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:17.450438 20418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:17.451327 20418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:17.452282 20418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:17.461341 20424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:17.461377 20426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:17.461408 20423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:17.461508 20418 server_base.cc:1034] running on GCE node
I20260619 11:05:17.461954 20418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:17.462620 20418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:17.463743 20418 hybrid_clock.cc:648] HybridClock initialized: now 1781867117463726 us; error 30 us; skew 500 ppm
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:17.466344 20418 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:17.467139 20418 webserver.cc:460] Webserver started at http://127.16.61.2:36941/ using document root <none> and password file <none>
I20260619 11:05:17.467720 20418 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:17.467805 20418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:17.468036 20418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:17.469894 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "6b0aeecb602e4092b324495f2d116e5a"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "0d17fe28de8374361475726f7d07706c"
server_key_iv: "7122b0eaf9f6e767dcb662439bf7da2e"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.470367 20418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "6b0aeecb602e4092b324495f2d116e5a"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "0d17fe28de8374361475726f7d07706c"
server_key_iv: "7122b0eaf9f6e767dcb662439bf7da2e"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.473747 20418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:05:17.475957 20433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.477088 20418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:05:17.477202 20418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "6b0aeecb602e4092b324495f2d116e5a"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "0d17fe28de8374361475726f7d07706c"
server_key_iv: "7122b0eaf9f6e767dcb662439bf7da2e"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.477300 20418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:17.509519 20418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:17.512439 20418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:17.512585 20418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:17.513069 20418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:17.513782 20418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:17.513826 20418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.513863 20418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:17.513878 20418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.522197 20418 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:37309
I20260619 11:05:17.522220 20546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:37309 every 8 connection(s)
I20260619 11:05:17.523018 20418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:17.531633 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 20418
I20260619 11:05:17.531778 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:17.532027 16628 external_mini_cluster.cc:1428] Setting key 273dd402f4a95e1c3e5f5845572d5a46
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:17.536423 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:17.524695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:57299 (local address 127.16.61.62:42603)
0619 11:05:17.524820 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:17.524824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:17.525625 (+ 801us) server_negotiation.cc:407] Connection header received
0619 11:05:17.526960 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:17.526963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:17.527019 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:17.527113 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:17.528301 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.528826 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.529941 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.530161 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.530269 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:17.532738 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:17.532741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:17.532743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:17.532789 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:17.534217 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.534804 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.534807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.534808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.534879 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.535272 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.535279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.535281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.535527 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:17.535634 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:17.536130 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 11:05:17.536260 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:05:17.537271 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:17.525052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42603 (local address 127.16.61.2:57299)
0619 11:05:17.525479 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:17.525519 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:17.526751 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:17.527309 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:17.527318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:17.527653 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:17.528155 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:17.528168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.528955 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.528959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:17.529793 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:17.529802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.530314 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.530318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:17.530481 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:17.530957 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:17.530973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:17.532549 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:17.534333 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:17.534339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:17.534353 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:17.534702 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:17.534965 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:17.534971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:17.534976 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:17.535163 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:17.535630 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:17.535635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:17.535894 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 11:05:17.536043 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 11:05:17.536263 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260619 11:05:17.538396 20547 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42603
I20260619 11:05:17.538631 20547 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:17.539064 20547 heartbeater.cc:511] Master 127.16.61.62:42603 requested a full tablet report, sending...
I20260619 11:05:17.540047 20222 ts_manager.cc:194] Registered new tserver with Master: 6b0aeecb602e4092b324495f2d116e5a (127.16.61.2:37309)
I20260619 11:05:17.540598 20222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:57299
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:05:17.559716 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42603
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:17.665803 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:17.666095 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:17.666167 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:17.670365 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:17.670446 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:17.670532 20552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:17.675624 20552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42603
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:17.676605 20552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:17.677554 20552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:17.686782 20557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:17.686798 20558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:17.686831 20560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:17.686965 20552 server_base.cc:1034] running on GCE node
I20260619 11:05:17.687376 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:17.688036 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:17.689214 20552 hybrid_clock.cc:648] HybridClock initialized: now 1781867117689198 us; error 31 us; skew 500 ppm
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:17.691959 20552 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:17.692795 20552 webserver.cc:460] Webserver started at http://127.16.61.3:41001/ using document root <none> and password file <none>
I20260619 11:05:17.693321 20552 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:17.693413 20552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:17.693611 20552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:17.695485 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "a50e3181bfac4d0285f5e63d62c01102"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "382122986e0cefbfe0cbcd94094597f9"
server_key_iv: "5e98f58ec61ce6d36f2a2220fe4932fd"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.696084 20552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "a50e3181bfac4d0285f5e63d62c01102"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "382122986e0cefbfe0cbcd94094597f9"
server_key_iv: "5e98f58ec61ce6d36f2a2220fe4932fd"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.699404 20552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:17.701632 20567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.702728 20552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:05:17.702857 20552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "a50e3181bfac4d0285f5e63d62c01102"
format_stamp: "Formatted at 2026-06-19 11:05:17 on dist-test-slave-8vcz"
server_key: "382122986e0cefbfe0cbcd94094597f9"
server_key_iv: "5e98f58ec61ce6d36f2a2220fe4932fd"
server_key_version: "encryptionkey@0"
I20260619 11:05:17.702966 20552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:17.715628 20552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:17.718680 20552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:17.718866 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:17.719404 20552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:17.720206 20552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:17.720273 20552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.720340 20552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:17.720386 20552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:17.728837 20552 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:43891
I20260619 11:05:17.728864 20680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:43891 every 8 connection(s)
I20260619 11:05:17.729713 20552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:17.735419 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 20552
I20260619 11:05:17.735539 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:17.735859 16628 external_mini_cluster.cc:1428] Setting key 120b08b24426c595cae1e7be236fbdd3
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:17.742923 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:17.731329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:52909 (local address 127.16.61.62:42603)
0619 11:05:17.731464 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:17.731468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:17.732276 (+ 808us) server_negotiation.cc:407] Connection header received
0619 11:05:17.733628 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:17.733631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:17.733689 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:17.733793 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:17.734928 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.735656 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.737053 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.737337 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.737421 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:17.739811 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:17.739814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:17.739817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:17.739839 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:17.741025 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.741442 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.741445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.741446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.741556 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.741883 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.741886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.741887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.742104 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:17.742213 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:17.742709 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 11:05:17.742811 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:05:17.743743 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:17.731643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42603 (local address 127.16.61.3:52909)
0619 11:05:17.732115 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:17.732161 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:17.733422 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:17.733959 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:17.733966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:17.734342 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:17.734773 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:17.734785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.735788 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.735792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:17.736936 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:17.736946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.737478 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.737482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:17.737678 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:17.738296 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:17.738312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:17.739611 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:17.741121 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:17.741125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:17.741135 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:17.741357 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:17.741642 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:17.741644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:17.741645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:17.741779 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:17.742245 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:17.742251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:17.742487 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 11:05:17.742651 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 11:05:17.742892 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260619 11:05:17.744807 20681 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42603
I20260619 11:05:17.745038 20681 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:17.745474 20681 heartbeater.cc:511] Master 127.16.61.62:42603 requested a full tablet report, sending...
I20260619 11:05:17.746390 20222 ts_manager.cc:194] Registered new tserver with Master: a50e3181bfac4d0285f5e63d62c01102 (127.16.61.3:43891)
I20260619 11:05:17.746944 20222 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:52909
I20260619 11:05:17.749300 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:05:17.774021 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:17.765070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.16.61.1:34751)
0619 11:05:17.765405 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:17.765418 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:17.765440 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:05:17.765486 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:17.765492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:17.765656 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:17.765769 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:17.766415 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.767182 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.768079 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.768442 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.768525 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:17.770737 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:17.770745 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:17.770767 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:17.770791 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:17.772463 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.772861 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.772866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.772867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.772939 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.773156 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.773158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.773159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.773387 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:17.773530 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:17.773674 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 11:05:17.773786 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":128,"threads_started":1}
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:17.784327 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:17.776569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.16.61.62:42603)
0619 11:05:17.776698 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:17.776704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:17.776760 (+ 56us) server_negotiation.cc:407] Connection header received
0619 11:05:17.776901 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:17.776905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:17.776955 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:17.777038 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:17.777611 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.778109 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.778863 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:17.779081 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:17.779189 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:17.781531 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:17.781534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:17.781536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:17.781557 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:17.782715 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.783091 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.783093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.783095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.783161 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:17.783408 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:17.783410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:17.783411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:17.783597 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:17.783754 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:17.784028 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 11:05:17.784117 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 11:05:17.784737 20222 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55766
I20260619 11:05:17.785408 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 20284
I20260619 11:05:17.790796 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 20418
I20260619 11:05:17.795852 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 20552
I20260619 11:05:17.800968 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 20190
2026-06-19T11:05:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2614 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20693](info): set up 2 sockets
Jun 19 11:05:17 dist-test-slave-8vcz krb5kdc[20693](info): commencing operation
krb5kdc: starting...
W20260619 11:05:18.839815 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:18 dist-test-slave-8vcz krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:18.956743 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:36615
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:36615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:19.061550 20706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:19.061828 20706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:19.061904 20706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:19.065801 20706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:19.065883 20706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:19.065922 20706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:19.065979 20706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:19.066018 20706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:19.071166 20706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:36615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:36615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:19.072157 20706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:19.073114 20706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:05:19.081058 20706 server_base.cc:1034] running on GCE node
W20260619 11:05:19.080999 20712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:19.081007 20711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:19.081030 20714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:19.081804 20706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:19.082821 20706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:19.083999 20706 hybrid_clock.cc:648] HybridClock initialized: now 1781867119083986 us; error 24 us; skew 500 ppm
Jun 19 11:05:19 dist-test-slave-8vcz krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:19.086671 20706 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:19.087373 20706 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:19.087473 20706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:19.087738 20706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:19.089649 20706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "8334f74d8c194ae18751d3b8861d445e"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "d52fcbf0b36f378e46c553d54a699d8e"
server_key_iv: "3dbfe1eba8c9800c5224d0210df8ffdd"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.090188 20706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "8334f74d8c194ae18751d3b8861d445e"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "d52fcbf0b36f378e46c553d54a699d8e"
server_key_iv: "3dbfe1eba8c9800c5224d0210df8ffdd"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.093705 20706 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 11:05:19.095952 20720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.097036 20706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:05:19.097174 20706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "8334f74d8c194ae18751d3b8861d445e"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "d52fcbf0b36f378e46c553d54a699d8e"
server_key_iv: "3dbfe1eba8c9800c5224d0210df8ffdd"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.097278 20706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:19.113700 20706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:19.116835 20706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:19.117033 20706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:19.123713 20706 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:36615
I20260619 11:05:19.123939 20772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:36615 every 8 connection(s)
I20260619 11:05:19.124356 20706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:19.126952 20773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:19.132015 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 20706
I20260619 11:05:19.132154 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:19.132403 16628 external_mini_cluster.cc:1428] Setting key ff05e1da99451da46cef79ff6043b7a4
I20260619 11:05:19.132436 20773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e: Bootstrap starting.
I20260619 11:05:19.135025 20773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:19.135880 20773 log.cc:824] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:19.137729 20773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e: No bootstrap required, opened a new log
Jun 19 11:05:19 dist-test-slave-8vcz krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:19.141906 20773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8334f74d8c194ae18751d3b8861d445e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36615 } }
I20260619 11:05:19.142132 20773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:19.142235 20773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8334f74d8c194ae18751d3b8861d445e, State: Initialized, Role: FOLLOWER
I20260619 11:05:19.142661 20773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8334f74d8c194ae18751d3b8861d445e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36615 } }
I20260619 11:05:19.142827 20773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:19.142930 20773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:19.143025 20773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:19.143903 20773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8334f74d8c194ae18751d3b8861d445e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36615 } }
I20260619 11:05:19.144209 20773 leader_election.cc:302] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8334f74d8c194ae18751d3b8861d445e; no voters:
I20260619 11:05:19.144495 20773 leader_election.cc:288] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:19.144603 20778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:19.144832 20778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [term 1 LEADER]: Becoming Leader. State: Replica: 8334f74d8c194ae18751d3b8861d445e, State: Running, Role: LEADER
I20260619 11:05:19.145159 20778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8334f74d8c194ae18751d3b8861d445e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36615 } }
I20260619 11:05:19.145764 20773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:19.147142 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.133684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.16.61.62:36615)
0619 11:05:19.134057 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.134075 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.134116 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:05:19.134894 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.134918 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.135347 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.135718 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:19.136479 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.137456 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.138592 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.139047 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.139179 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.141902 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:19.141907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:19.141919 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:19.141943 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:19.143871 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.144318 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.144326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.144331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.144478 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.144868 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.144872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.144873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.145330 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:19.145507 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.145949 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.146261 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260619 11:05:19.148486 20779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8334f74d8c194ae18751d3b8861d445e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8334f74d8c194ae18751d3b8861d445e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36615 } } }
I20260619 11:05:19.149595 20779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:19.148536 20780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8334f74d8c194ae18751d3b8861d445e. Latest consensus state: current_term: 1 leader_uuid: "8334f74d8c194ae18751d3b8861d445e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8334f74d8c194ae18751d3b8861d445e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36615 } } }
I20260619 11:05:19.149855 20780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e [sys.catalog]: This master's current role is: LEADER
W20260619 11:05:19.150810 20793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:05:19.150900 20793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:05:19.150977 20795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:19.151806 20795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:19.156793 20795 catalog_manager.cc:1347] Generated new cluster ID: 6a8349c94aa94db7ab8eaa28551b6dfe
I20260619 11:05:19.156886 20795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:19.168038 20795 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:19.169183 20795 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:19.177078 20795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e: Generated new TSK 0
I20260619 11:05:19.177781 20795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:19.211000 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:36615
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:19.317266 20799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:19.317495 20799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:19.317538 20799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:19.321498 20799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:19.321547 20799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:19.321638 20799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:19.326794 20799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:19.327782 20799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:19.328720 20799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:19.338024 20807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:19.338070 20804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:19.338027 20805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:19.338395 20799 server_base.cc:1034] running on GCE node
I20260619 11:05:19.338781 20799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:19.339357 20799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:19.340518 20799 hybrid_clock.cc:648] HybridClock initialized: now 1781867119340498 us; error 30 us; skew 500 ppm
Jun 19 11:05:19 dist-test-slave-8vcz krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:19.343111 20799 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:19.343811 20799 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:19.343878 20799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:19.344031 20799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:19.345868 20799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "2fbb2288cbc443988cfa7c1de07d0f94"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "0257360c1c2d89a1bbbf271afbc26f1b"
server_key_iv: "3f94d0165b0635c3fda4047819088609"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.346490 20799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "2fbb2288cbc443988cfa7c1de07d0f94"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "0257360c1c2d89a1bbbf271afbc26f1b"
server_key_iv: "3f94d0165b0635c3fda4047819088609"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.350034 20799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:19.352320 20813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.353559 20799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:19.353681 20799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "2fbb2288cbc443988cfa7c1de07d0f94"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "0257360c1c2d89a1bbbf271afbc26f1b"
server_key_iv: "3f94d0165b0635c3fda4047819088609"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.353781 20799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:19.370101 20799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:19.372869 20799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:19.373055 20799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:19.373596 20799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:19.374351 20799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:19.374393 20799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.374454 20799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:19.374485 20799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.382820 20799 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:42441
I20260619 11:05:19.382838 20926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:42441 every 8 connection(s)
I20260619 11:05:19.383370 20799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:19.386584 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 20799
I20260619 11:05:19.386718 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:19.386978 16628 external_mini_cluster.cc:1428] Setting key 287d1c263607a38b91950d30d1e84531
Jun 19 11:05:19 dist-test-slave-8vcz krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:05:19.397198 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.384969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:47267 (local address 127.16.61.62:36615)
0619 11:05:19.385096 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.385100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.385870 (+ 770us) server_negotiation.cc:407] Connection header received
0619 11:05:19.387308 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.387311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.387367 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.387477 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:19.388638 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.389154 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.390755 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.390982 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.391064 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.393743 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:19.393752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:19.393754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:19.393775 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:19.395151 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.395632 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.395637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.395639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.395781 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.396146 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.396149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.396151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.396327 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:19.396437 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.396922 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.397036 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:19.397994 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.385330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:36615 (local address 127.16.61.1:47267)
0619 11:05:19.385738 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:19.385776 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:19.387094 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:19.387641 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:19.387647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:19.388042 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:19.388500 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.388513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.389277 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.389283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.390628 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.390642 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.391131 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.391137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.391365 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.391988 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:19.392012 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:19.393585 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:19.395269 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:19.395275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:19.395288 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:19.395531 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:19.395873 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:19.395877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:19.395879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:19.396012 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:19.396433 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:19.396440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:19.396666 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 11:05:19.396850 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 11:05:19.397141 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
I20260619 11:05:19.399143 20927 heartbeater.cc:347] Connected to a master server at 127.16.61.62:36615
I20260619 11:05:19.399358 20927 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:19.399832 20927 heartbeater.cc:511] Master 127.16.61.62:36615 requested a full tablet report, sending...
I20260619 11:05:19.401288 20737 ts_manager.cc:194] Registered new tserver with Master: 2fbb2288cbc443988cfa7c1de07d0f94 (127.16.61.1:42441)
I20260619 11:05:19.402401 20737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:47267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:19.414382 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:36615
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:19.521116 20932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:19.521353 20932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:19.521402 20932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:19.525326 20932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:19.525382 20932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:19.525439 20932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:19.530539 20932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:19.531535 20932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:19.532451 20932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:19.541503 20938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:19.541512 20940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:19.541580 20937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:19.541651 20932 server_base.cc:1034] running on GCE node
I20260619 11:05:19.542169 20932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:19.542748 20932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:19.543922 20932 hybrid_clock.cc:648] HybridClock initialized: now 1781867119543888 us; error 45 us; skew 500 ppm
Jun 19 11:05:19 dist-test-slave-8vcz krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:19.547035 20932 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:19.547851 20932 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:19.547919 20932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:19.548077 20932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:19.549952 20932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "93a49ed128c0456d9409de0ae585af72"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "f8def646d3022cebcfa0371c8b442617"
server_key_iv: "5a2793ba231608a2515183ba7478fd49"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.550431 20932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "93a49ed128c0456d9409de0ae585af72"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "f8def646d3022cebcfa0371c8b442617"
server_key_iv: "5a2793ba231608a2515183ba7478fd49"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.554006 20932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:19.556320 20946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.557513 20932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:19.557638 20932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "93a49ed128c0456d9409de0ae585af72"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "f8def646d3022cebcfa0371c8b442617"
server_key_iv: "5a2793ba231608a2515183ba7478fd49"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.557747 20932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:19.577121 20932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:19.579823 20932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:19.580004 20932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:19.580539 20932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:19.581332 20932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:19.581377 20932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.581444 20932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:19.581475 20932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.589879 20932 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:42015
I20260619 11:05:19.589951 21059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:42015 every 8 connection(s)
I20260619 11:05:19.590449 20932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:05:19 dist-test-slave-8vcz krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:19.599994 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 20932
I20260619 11:05:19.600119 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:19.600445 16628 external_mini_cluster.cc:1428] Setting key d2f4dc6cf92806c1e58a1d36a16e0c3d
I20260619 11:05:19.603811 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.592353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:45331 (local address 127.16.61.62:36615)
0619 11:05:19.592502 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.592506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.593693 (+ 1187us) server_negotiation.cc:407] Connection header received
0619 11:05:19.594481 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.594483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.594528 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.594601 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:19.595824 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.596358 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.597464 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.597672 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.597757 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.599997 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:19.600001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:19.600005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:19.600037 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:19.601658 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.602220 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.602224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.602226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.602347 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.602655 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.602661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.602666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.602911 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:19.603015 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.603459 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.603594 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:05:19.604781 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.592754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:36615 (local address 127.16.61.2:45331)
0619 11:05:19.593555 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:19.593592 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:19.594300 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:19.594758 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:19.594764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:19.595080 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:19.595639 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.595656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.596479 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.596481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.597339 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.597348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.597798 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.597800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.597928 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.598412 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:19.598426 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:19.599833 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:19.601794 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:19.601801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:19.601815 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:19.602120 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:19.602442 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:19.602444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:19.602445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:19.602562 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:19.603011 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:19.603016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:19.603217 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 11:05:19.603374 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 11:05:19.603737 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":110,"threads_started":1}
I20260619 11:05:19.605907 21060 heartbeater.cc:347] Connected to a master server at 127.16.61.62:36615
I20260619 11:05:19.606195 21060 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:19.606639 21060 heartbeater.cc:511] Master 127.16.61.62:36615 requested a full tablet report, sending...
I20260619 11:05:19.607702 20737 ts_manager.cc:194] Registered new tserver with Master: 93a49ed128c0456d9409de0ae585af72 (127.16.61.2:42015)
I20260619 11:05:19.608256 20737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:45331
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:05:19.627115 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:36615
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:19.733485 21065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:19.733757 21065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:19.733844 21065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:19.737964 21065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:19.738036 21065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:19.738097 21065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:19.743155 21065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:36615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:19.744066 21065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:19.744925 21065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:19.753911 21073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:19.753870 21070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:19.753870 21071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:19.754046 21065 server_base.cc:1034] running on GCE node
I20260619 11:05:19.754474 21065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:19.755045 21065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:19.756245 21065 hybrid_clock.cc:648] HybridClock initialized: now 1781867119756221 us; error 33 us; skew 500 ppm
Jun 19 11:05:19 dist-test-slave-8vcz krb5kdc[20693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:19.759099 21065 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:19.759900 21065 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:19.759987 21065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:19.760202 21065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:19.762264 21065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "792897173fbfb9f36e7a13b63a0909f8"
server_key_iv: "8fb7f3826a321a8d1e1b2461e60db8b8"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.762836 21065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "792897173fbfb9f36e7a13b63a0909f8"
server_key_iv: "8fb7f3826a321a8d1e1b2461e60db8b8"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.766304 21065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:05:19.768687 21079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.769899 21065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:19.770037 21065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e"
format_stamp: "Formatted at 2026-06-19 11:05:19 on dist-test-slave-8vcz"
server_key: "792897173fbfb9f36e7a13b63a0909f8"
server_key_iv: "8fb7f3826a321a8d1e1b2461e60db8b8"
server_key_version: "encryptionkey@0"
I20260619 11:05:19.770145 21065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:19.779775 21065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:19.782775 21065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:19.782965 21065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:19.783473 21065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:19.784256 21065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:19.784330 21065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.784399 21065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:19.784453 21065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:19.792646 21065 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:32837
I20260619 11:05:19.792662 21192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:32837 every 8 connection(s)
I20260619 11:05:19.793200 21065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:05:19 dist-test-slave-8vcz krb5kdc[20693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867119, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:19.802471 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 21065
I20260619 11:05:19.802589 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:19.802901 16628 external_mini_cluster.cc:1428] Setting key 5302bd3d159593d94450399c102323d2
I20260619 11:05:19.805789 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.794774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:34833 (local address 127.16.61.62:36615)
0619 11:05:19.794902 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.794906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.795981 (+ 1075us) server_negotiation.cc:407] Connection header received
0619 11:05:19.796758 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.796760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.796818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.796892 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:19.797937 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.798475 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.799527 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.799830 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.799919 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.802198 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:19.802201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:19.802203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:19.802223 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:19.803819 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.804376 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.804385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.804386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.804456 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.804830 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.804836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.804841 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.805070 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:19.805154 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.805583 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.805671 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:05:19.806565 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.795067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:36615 (local address 127.16.61.3:34833)
0619 11:05:19.795841 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:19.795880 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:19.796585 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:19.797046 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:19.797052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:19.797361 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:19.797804 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.797815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.798574 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.798576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.799406 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.799413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.800013 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.800016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.800151 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.800619 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:19.800634 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:19.802059 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:19.803912 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:19.803916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:19.803926 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:19.804284 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:19.804545 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:19.804548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:19.804550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:19.804714 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:19.805171 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:19.805176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:19.805391 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 11:05:19.805540 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 11:05:19.805781 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260619 11:05:19.807626 21193 heartbeater.cc:347] Connected to a master server at 127.16.61.62:36615
I20260619 11:05:19.807842 21193 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:19.808195 21193 heartbeater.cc:511] Master 127.16.61.62:36615 requested a full tablet report, sending...
I20260619 11:05:19.809062 20737 ts_manager.cc:194] Registered new tserver with Master: a4d49aa6f74943de826e2e9d8f4b6a1e (127.16.61.3:32837)
I20260619 11:05:19.809584 20737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:34833
I20260619 11:05:19.816533 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:05:19.824692 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.818283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.16.61.62:36615)
0619 11:05:19.818397 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.818400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.818494 (+ 94us) server_negotiation.cc:407] Connection header received
0619 11:05:19.818591 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.818593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.818627 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.818692 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:19.819304 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.819869 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.820644 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.820860 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.820949 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.822091 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:19.822094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:19.822096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:19.822115 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:19.823402 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.823799 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.823802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.823803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.823873 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:19.824071 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:19.824072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:19.824073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:19.824242 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:19.824337 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.824507 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.824586 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:05:19.827337 20737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:19.830147 20737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:19.843308 21204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.837598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:32837 (local address 127.0.0.1:53604)
0619 11:05:19.837962 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:19.838006 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:19.838115 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:19.838883 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:19.838890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:19.838912 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:19.839036 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.839043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.840541 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.840545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.841938 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.841948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.842943 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.842946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.843025 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.843044 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:05:19.843110 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:05:19.843162 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260619 11:05:19.843400 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.837718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.16.61.3:32837)
0619 11:05:19.838403 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.838415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.838436 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:05:19.838505 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.838511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.838708 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.838900 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:19.839159 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.840411 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.842069 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.842826 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.842938 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.843054 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.843137 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.843241 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":137,"threads_started":1}
I20260619 11:05:19.843631 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.838285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42441 (local address 127.0.0.1:38074)
0619 11:05:19.839141 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:19.839153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:19.839224 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:19.839858 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:19.839861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:19.839877 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:19.839961 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.839966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.841115 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.841119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.842229 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.842236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.843263 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.843267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.843352 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.843376 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:05:19.843453 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:05:19.843531 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":82,"threads_started":1}
I20260619 11:05:19.843194 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.837861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:42015 (local address 127.0.0.1:42256)
0619 11:05:19.838832 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:19.838845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:19.838944 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:19.839419 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:19.839422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:19.839438 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:19.839532 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.839538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.840814 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.840819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.841916 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.841924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.842669 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.842672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.842770 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.842804 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:05:19.842932 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 11:05:19.843035 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":21,"spinlock_wait_cycles":2304,"thread_start_us":80,"threads_started":1}
I20260619 11:05:19.844022 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.838347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.16.61.2:42015)
0619 11:05:19.838692 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.838705 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.839029 (+ 324us) server_negotiation.cc:407] Connection header received
0619 11:05:19.839088 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.839091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.839279 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.839421 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:19.839647 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.840689 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.842030 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.842552 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.842668 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.842808 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.843655 (+ 847us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.843760 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":80,"threads_started":1}
I20260619 11:05:19.844489 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.838663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.16.61.1:42441)
0619 11:05:19.839442 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.839452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.839471 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:05:19.839537 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.839542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.839730 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.839887 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:19.840051 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.840976 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.842360 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.843131 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.843259 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.843374 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.843911 (+ 537us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.844144 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":108,"threads_started":1}
I20260619 11:05:19.846359 21127 tablet_service.cc:1467] Processing CreateTablet for tablet beaa4a8cfc8144bf8b8ce78a3224196f (DEFAULT_TABLE table=test-table [id=58d669e671824620b7555ec60bfdc977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:19.846683 20994 tablet_service.cc:1467] Processing CreateTablet for tablet beaa4a8cfc8144bf8b8ce78a3224196f (DEFAULT_TABLE table=test-table [id=58d669e671824620b7555ec60bfdc977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:19.846763 20861 tablet_service.cc:1467] Processing CreateTablet for tablet beaa4a8cfc8144bf8b8ce78a3224196f (DEFAULT_TABLE table=test-table [id=58d669e671824620b7555ec60bfdc977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:19.847465 21127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaa4a8cfc8144bf8b8ce78a3224196f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:19.847584 20994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaa4a8cfc8144bf8b8ce78a3224196f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:19.847610 20861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beaa4a8cfc8144bf8b8ce78a3224196f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:19.853259 21210 tablet_bootstrap.cc:492] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94: Bootstrap starting.
I20260619 11:05:19.854554 21211 tablet_bootstrap.cc:492] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: Bootstrap starting.
I20260619 11:05:19.855336 21210 tablet_bootstrap.cc:654] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:19.855959 21212 tablet_bootstrap.cc:492] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: Bootstrap starting.
I20260619 11:05:19.856372 21211 tablet_bootstrap.cc:654] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:19.856470 21210 log.cc:824] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:19.857539 21211 log.cc:824] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:19.858151 21212 tablet_bootstrap.cc:654] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:19.858433 21210 tablet_bootstrap.cc:492] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94: No bootstrap required, opened a new log
I20260619 11:05:19.858618 21210 ts_tablet_manager.cc:1397] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 11:05:19.859158 21212 log.cc:824] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:19.859416 21211 tablet_bootstrap.cc:492] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: No bootstrap required, opened a new log
I20260619 11:05:19.859591 21211 ts_tablet_manager.cc:1397] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:05:19.860958 21212 tablet_bootstrap.cc:492] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: No bootstrap required, opened a new log
I20260619 11:05:19.861124 21212 ts_tablet_manager.cc:1397] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:05:19.862260 21210 raft_consensus.cc:348] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.862500 21210 raft_consensus.cc:374] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:19.862545 21210 raft_consensus.cc:729] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbb2288cbc443988cfa7c1de07d0f94, State: Initialized, Role: FOLLOWER
I20260619 11:05:19.862686 21211 raft_consensus.cc:348] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.862963 21211 raft_consensus.cc:374] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:19.863029 21211 raft_consensus.cc:729] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d49aa6f74943de826e2e9d8f4b6a1e, State: Initialized, Role: FOLLOWER
I20260619 11:05:19.862963 21210 consensus_queue.cc:260] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.863440 21211 consensus_queue.cc:260] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.863610 20927 heartbeater.cc:503] Master 127.16.61.62:36615 was elected leader, sending a full tablet report...
I20260619 11:05:19.864017 21210 ts_tablet_manager.cc:1428] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 11:05:19.864351 21211 ts_tablet_manager.cc:1428] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:05:19.864668 21193 heartbeater.cc:503] Master 127.16.61.62:36615 was elected leader, sending a full tablet report...
I20260619 11:05:19.864593 21212 raft_consensus.cc:348] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.864873 21212 raft_consensus.cc:374] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:19.864940 21212 raft_consensus.cc:729] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a49ed128c0456d9409de0ae585af72, State: Initialized, Role: FOLLOWER
I20260619 11:05:19.865310 21212 consensus_queue.cc:260] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.866050 21212 ts_tablet_manager.cc:1428] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:05:19.866385 21060 heartbeater.cc:503] Master 127.16.61.62:36615 was elected leader, sending a full tablet report...
W20260619 11:05:19.884366 20928 tablet.cc:2369] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:19.954919 21217 raft_consensus.cc:482] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:19.955123 21217 raft_consensus.cc:504] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.956338 21217 leader_election.cc:288] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fbb2288cbc443988cfa7c1de07d0f94 (127.16.61.1:42441), 93a49ed128c0456d9409de0ae585af72 (127.16.61.2:42015)
I20260619 11:05:19.960366 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.956598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:42015 (local address 127.16.61.3:46823)
0619 11:05:19.956695 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:19.956710 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:19.956803 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:19.957101 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:19.957105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:19.957124 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:19.957239 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.957244 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.958043 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.958049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.959312 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.959321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.960070 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.960072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.960145 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.960164 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:05:19.960223 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 11:05:19.960271 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 11:05:19.960479 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.956688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:46823 (local address 127.16.61.2:42015)
0619 11:05:19.956801 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.956806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.956824 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:05:19.956949 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.956953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.957003 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.957088 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:19.957379 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.957930 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.959428 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.959962 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.960064 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.960125 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.960252 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.960319 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:05:19.960939 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.956679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:45615 (local address 127.16.61.1:42441)
0619 11:05:19.956802 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.956806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.957136 (+ 330us) server_negotiation.cc:407] Connection header received
0619 11:05:19.957281 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.957285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.957357 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.957431 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:19.957679 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.958254 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.959757 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.960337 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.960437 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.960491 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.960745 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.960817 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:19.960978 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.956598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42441 (local address 127.16.61.3:45615)
0619 11:05:19.957046 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:19.957057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:19.957171 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:19.957438 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:19.957441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:19.957455 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:19.957576 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.957581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.958528 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.958531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.959621 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:19.959628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.960469 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.960472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:19.960586 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.960620 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:05:19.960701 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:05:19.960839 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":105,"thread_start_us":159,"threads_started":1}
I20260619 11:05:19.961140 21014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaa4a8cfc8144bf8b8ce78a3224196f" candidate_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a49ed128c0456d9409de0ae585af72" is_pre_election: true
I20260619 11:05:19.961458 21014 raft_consensus.cc:2454] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d49aa6f74943de826e2e9d8f4b6a1e in term 0.
I20260619 11:05:19.961613 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaa4a8cfc8144bf8b8ce78a3224196f" candidate_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" is_pre_election: true
I20260619 11:05:19.961877 20881 raft_consensus.cc:2454] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d49aa6f74943de826e2e9d8f4b6a1e in term 0.
I20260619 11:05:19.961997 21082 leader_election.cc:302] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a49ed128c0456d9409de0ae585af72, a4d49aa6f74943de826e2e9d8f4b6a1e; no voters:
I20260619 11:05:19.962266 21217 raft_consensus.cc:2781] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:19.962350 21217 raft_consensus.cc:482] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:19.962392 21217 raft_consensus.cc:3037] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:19.963354 21217 raft_consensus.cc:504] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.963650 21217 leader_election.cc:288] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [CANDIDATE]: Term 1 election: Requested vote from peers 2fbb2288cbc443988cfa7c1de07d0f94 (127.16.61.1:42441), 93a49ed128c0456d9409de0ae585af72 (127.16.61.2:42015)
I20260619 11:05:19.963891 20881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaa4a8cfc8144bf8b8ce78a3224196f" candidate_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fbb2288cbc443988cfa7c1de07d0f94"
I20260619 11:05:19.963994 21014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beaa4a8cfc8144bf8b8ce78a3224196f" candidate_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a49ed128c0456d9409de0ae585af72"
I20260619 11:05:19.964020 20881 raft_consensus.cc:3037] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:19.964118 21014 raft_consensus.cc:3037] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:19.964998 20881 raft_consensus.cc:2454] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d49aa6f74943de826e2e9d8f4b6a1e in term 1.
I20260619 11:05:19.965114 21014 raft_consensus.cc:2454] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d49aa6f74943de826e2e9d8f4b6a1e in term 1.
I20260619 11:05:19.965277 21081 leader_election.cc:302] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbb2288cbc443988cfa7c1de07d0f94, a4d49aa6f74943de826e2e9d8f4b6a1e; no voters:
I20260619 11:05:19.965449 21217 raft_consensus.cc:2781] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:19.965638 21217 raft_consensus.cc:686] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 1 LEADER]: Becoming Leader. State: Replica: a4d49aa6f74943de826e2e9d8f4b6a1e, State: Running, Role: LEADER
I20260619 11:05:19.965934 21217 consensus_queue.cc:237] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } }
I20260619 11:05:19.968936 20737 catalog_manager.cc:5495] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e reported cstate change: term changed from 0 to 1, leader changed from <none> to a4d49aa6f74943de826e2e9d8f4b6a1e (127.16.61.3). New cstate: current_term: 1 leader_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d49aa6f74943de826e2e9d8f4b6a1e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:19.996001 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:19.992616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.16.61.3:32837)
0619 11:05:19.992739 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:19.992743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:19.992762 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:05:19.992903 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:19.992905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:19.992954 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:19.993041 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:19.993253 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.993732 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.994558 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:19.994767 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:19.994869 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:19.995168 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:19.995573 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:19.995683 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:19.995847 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 11:05:19.995900 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:20.001672 20881 raft_consensus.cc:1261] T beaa4a8cfc8144bf8b8ce78a3224196f P 2fbb2288cbc443988cfa7c1de07d0f94 [term 1 FOLLOWER]: Refusing update from remote peer a4d49aa6f74943de826e2e9d8f4b6a1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:20.001672 21014 raft_consensus.cc:1261] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 1 FOLLOWER]: Refusing update from remote peer a4d49aa6f74943de826e2e9d8f4b6a1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:20.002501 21220 consensus_queue.cc:1035] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fbb2288cbc443988cfa7c1de07d0f94" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:20.002709 21217 consensus_queue.cc:1035] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a49ed128c0456d9409de0ae585af72" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:20.009133 21223 mvcc.cc:205] Tried to move back new op lower bound from 7298527723519164416 to 7298527723382808576. Current Snapshot: MvccSnapshot[applied={T|T < 7298527723519164416}]
I20260619 11:05:20.012801 21225 mvcc.cc:205] Tried to move back new op lower bound from 7298527723519164416 to 7298527723382808576. Current Snapshot: MvccSnapshot[applied={T|T < 7298527723519164416}]
I20260619 11:05:20.018270 20737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:20.018448 20737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:20.020915 20737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8334f74d8c194ae18751d3b8861d445e: Sending DeleteTablet for 3 replicas of tablet beaa4a8cfc8144bf8b8ce78a3224196f
I20260619 11:05:20.021654 20994 tablet_service.cc:1514] Processing DeleteTablet for tablet beaa4a8cfc8144bf8b8ce78a3224196f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:20 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:42256
I20260619 11:05:20.021689 21127 tablet_service.cc:1514] Processing DeleteTablet for tablet beaa4a8cfc8144bf8b8ce78a3224196f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:20 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:53604
I20260619 11:05:20.021838 20861 tablet_service.cc:1514] Processing DeleteTablet for tablet beaa4a8cfc8144bf8b8ce78a3224196f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:20 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:38074
I20260619 11:05:20.022169 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 20799
I20260619 11:05:20.022389 21233 tablet_replica.cc:330] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: stopping tablet replica
I20260619 11:05:20.022706 21233 raft_consensus.cc:2229] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:20.023047 21233 raft_consensus.cc:2258] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:20.024561 21233 ts_tablet_manager.cc:1905] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:20.027976 21233 ts_tablet_manager.cc:1918] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:20.028077 21233 log.cc:1196] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/beaa4a8cfc8144bf8b8ce78a3224196f
I20260619 11:05:20.028313 21234 tablet_replica.cc:330] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: stopping tablet replica
I20260619 11:05:20.028465 21233 ts_tablet_manager.cc:1939] T beaa4a8cfc8144bf8b8ce78a3224196f P 93a49ed128c0456d9409de0ae585af72: Deleting consensus metadata
I20260619 11:05:20.028638 21234 raft_consensus.cc:2229] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:20.029055 21234 raft_consensus.cc:2258] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:20.029853 20723 catalog_manager.cc:4841] TS 93a49ed128c0456d9409de0ae585af72 (127.16.61.2:42015): tablet beaa4a8cfc8144bf8b8ce78a3224196f (table test-table [id=58d669e671824620b7555ec60bfdc977]) successfully deleted
W20260619 11:05:20.029994 20722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:42441 (error 108)
I20260619 11:05:20.030066 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 20932
I20260619 11:05:20.031615 21234 ts_tablet_manager.cc:1905] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:20.034158 21234 ts_tablet_manager.cc:1918] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:20.034251 21234 log.cc:1196] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/beaa4a8cfc8144bf8b8ce78a3224196f
I20260619 11:05:20.034598 21234 ts_tablet_manager.cc:1939] T beaa4a8cfc8144bf8b8ce78a3224196f P a4d49aa6f74943de826e2e9d8f4b6a1e: Deleting consensus metadata
I20260619 11:05:20.035722 20722 catalog_manager.cc:4841] TS a4d49aa6f74943de826e2e9d8f4b6a1e (127.16.61.3:32837): tablet beaa4a8cfc8144bf8b8ce78a3224196f (table test-table [id=58d669e671824620b7555ec60bfdc977]) successfully deleted
I20260619 11:05:20.036331 21206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.036044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42441 (local address 127.0.0.1:38078)
0619 11:05:20.036182 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:20.036250 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:42441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 11:05:20.036623 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 21065
W20260619 11:05:20.036626 20722 catalog_manager.cc:4584] TS 2fbb2288cbc443988cfa7c1de07d0f94 (127.16.61.1:42441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet beaa4a8cfc8144bf8b8ce78a3224196f: Network error: Client connection negotiation failed: client connection to 127.16.61.1:42441: connect: Connection refused (error 111)
I20260619 11:05:20.044184 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 20706
2026-06-19T11:05:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2242 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T11:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:20Z Disabled control of system clock
I20260619 11:05:20.073604 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:40103
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:40103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 11:05:20.180356 21240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:20.180572 21240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:20.180608 21240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:20.184492 21240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:20.184540 21240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:20.184554 21240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:20.184597 21240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:20.184631 21240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:20.189246 21240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:40103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:40103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:20.190150 21240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:20.191403 21240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:20.199294 21248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:20.199515 21246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:20.199407 21240 server_base.cc:1034] running on GCE node
W20260619 11:05:20.199297 21245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:20.200106 21240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:20.201077 21240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:20.202268 21240 hybrid_clock.cc:648] HybridClock initialized: now 1781867120202258 us; error 27 us; skew 500 ppm
I20260619 11:05:20.204000 21240 webserver.cc:460] Webserver started at http://127.16.61.62:45041/ using document root <none> and password file <none>
I20260619 11:05:20.204540 21240 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:20.204628 21240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:20.204841 21240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:20.206732 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "2f51f5b139c54e0cb7fc75bdf96924ec"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "7e4603baa663c89bb8be508af0e161bb"
server_key_iv: "c8b8d331faa373661d19a7771c6706cf"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.207324 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "2f51f5b139c54e0cb7fc75bdf96924ec"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "7e4603baa663c89bb8be508af0e161bb"
server_key_iv: "c8b8d331faa373661d19a7771c6706cf"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.210734 21240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:05:20.212944 21254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.214082 21240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:20.214214 21240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "2f51f5b139c54e0cb7fc75bdf96924ec"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "7e4603baa663c89bb8be508af0e161bb"
server_key_iv: "c8b8d331faa373661d19a7771c6706cf"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.214320 21240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:20.218631 21240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:20.219163 21240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:20.219336 21240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:20.225687 21240 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:40103
I20260619 11:05:20.225695 21306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:40103 every 8 connection(s)
I20260619 11:05:20.226523 21240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:20.228821 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 21240
I20260619 11:05:20.228907 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:20.229166 16628 external_mini_cluster.cc:1428] Setting key 546c29908c49e2b192947aa0dacb4b91
I20260619 11:05:20.229568 21307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:20.234784 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.230454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.16.61.62:40103)
0619 11:05:20.231094 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:20.231113 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:20.231153 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:05:20.232148 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:20.232166 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:20.232496 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:20.232824 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:20.233649 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:20.233654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:20.233665 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:20.233675 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:20.233724 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:20.233739 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:20.233881 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:20.233949 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 11:05:20.234174 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260619 11:05:20.235985 21307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec: Bootstrap starting.
I20260619 11:05:20.238054 21307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:20.238739 21307 log.cc:824] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:20.240758 21307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec: No bootstrap required, opened a new log
I20260619 11:05:20.244081 21307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51f5b139c54e0cb7fc75bdf96924ec" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40103 } }
I20260619 11:05:20.244278 21307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:20.244364 21307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f51f5b139c54e0cb7fc75bdf96924ec, State: Initialized, Role: FOLLOWER
I20260619 11:05:20.244771 21307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51f5b139c54e0cb7fc75bdf96924ec" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40103 } }
I20260619 11:05:20.244918 21307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:20.244982 21307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:20.245096 21307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:20.245930 21307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51f5b139c54e0cb7fc75bdf96924ec" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40103 } }
I20260619 11:05:20.246299 21307 leader_election.cc:302] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f51f5b139c54e0cb7fc75bdf96924ec; no voters:
I20260619 11:05:20.246601 21307 leader_election.cc:288] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:20.246666 21312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:20.246923 21312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [term 1 LEADER]: Becoming Leader. State: Replica: 2f51f5b139c54e0cb7fc75bdf96924ec, State: Running, Role: LEADER
I20260619 11:05:20.247278 21312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51f5b139c54e0cb7fc75bdf96924ec" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40103 } }
I20260619 11:05:20.247859 21307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:20.249114 21314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f51f5b139c54e0cb7fc75bdf96924ec. Latest consensus state: current_term: 1 leader_uuid: "2f51f5b139c54e0cb7fc75bdf96924ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51f5b139c54e0cb7fc75bdf96924ec" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40103 } } }
I20260619 11:05:20.249132 21313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f51f5b139c54e0cb7fc75bdf96924ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f51f5b139c54e0cb7fc75bdf96924ec" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 40103 } } }
I20260619 11:05:20.249241 21314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:20.249249 21313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:20.249605 21321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:20.252125 21321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:20.257382 21321 catalog_manager.cc:1347] Generated new cluster ID: 700411556a4d49fca99f0593ad4fb826
I20260619 11:05:20.257458 21321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:20.263978 21321 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:20.264257 21321 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:20.269941 21321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec: Generated new TSK 0
I20260619 11:05:20.270524 21321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:05:20.273347 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40103
--builtin_ntp_servers=127.16.61.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 11:05:20.382143 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:20.382361 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:20.382398 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:20.386379 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:20.386431 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:20.386513 21331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:20.391112 21331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:20.392089 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:20.393354 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:20.402128 21337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:20.402179 21336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:20.402328 21331 server_base.cc:1034] running on GCE node
W20260619 11:05:20.402531 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:20.402894 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:20.403501 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:20.404688 21331 hybrid_clock.cc:648] HybridClock initialized: now 1781867120404670 us; error 55 us; skew 500 ppm
I20260619 11:05:20.406481 21331 webserver.cc:460] Webserver started at http://127.16.61.1:43379/ using document root <none> and password file <none>
I20260619 11:05:20.407022 21331 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:20.407109 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:20.407342 21331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:20.409415 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "9aa8d04248b342b0821a4b06f374445b"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "47750acebc86d4b402d09ad30330f5cc"
server_key_iv: "9cd9405af171331e71391561f97eb288"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.409952 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "9aa8d04248b342b0821a4b06f374445b"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "47750acebc86d4b402d09ad30330f5cc"
server_key_iv: "9cd9405af171331e71391561f97eb288"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.413460 21331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:05:20.415835 21345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.417012 21331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:20.417152 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "9aa8d04248b342b0821a4b06f374445b"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "47750acebc86d4b402d09ad30330f5cc"
server_key_iv: "9cd9405af171331e71391561f97eb288"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.417263 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:20.421711 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:20.422266 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:20.422444 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:20.422936 21331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:20.423807 21331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:20.423878 21331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.423946 21331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:20.423978 21331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.432178 21331 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:35099
I20260619 11:05:20.432205 21458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:35099 every 8 connection(s)
I20260619 11:05:20.432960 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:20.438407 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.434582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:52761 (local address 127.16.61.62:40103)
0619 11:05:20.434714 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:20.434717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:20.435619 (+ 902us) server_negotiation.cc:407] Connection header received
0619 11:05:20.436856 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:20.436860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:20.436905 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:20.437011 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:20.437958 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:20.437962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:20.437964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:20.437977 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:20.438002 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:20.438010 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:20.438100 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:20.438242 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 11:05:20.438282 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:20.439011 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 21331
I20260619 11:05:20.439133 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:20.439239 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.434883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40103 (local address 127.16.61.1:52761)
0619 11:05:20.435479 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:20.435516 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:20.436659 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:20.437163 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:20.437169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:20.437247 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:20.437717 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:20.437718 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:20.437752 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:20.437810 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:20.438094 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:20.438102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:20.438110 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:05:20.438279 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 11:05:20.438493 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260619 11:05:20.439473 16628 external_mini_cluster.cc:1428] Setting key 6d5f20e496acfe9e28fab0f9291adfe6
I20260619 11:05:20.440387 21459 heartbeater.cc:347] Connected to a master server at 127.16.61.62:40103
I20260619 11:05:20.440587 21459 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:20.440965 21459 heartbeater.cc:511] Master 127.16.61.62:40103 requested a full tablet report, sending...
I20260619 11:05:20.441471 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40103
--builtin_ntp_servers=127.16.61.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 11:05:20.442462 21271 ts_manager.cc:194] Registered new tserver with Master: 9aa8d04248b342b0821a4b06f374445b (127.16.61.1:35099)
W20260619 11:05:20.547573 21462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:20.547811 21462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:20.547884 21462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:20.551915 21462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:20.551960 21462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:20.552012 21462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:20.556526 21462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:20.557457 21462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:20.558707 21462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:20.567545 21467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:20.567485 21468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:20.567485 21470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:20.567829 21462 server_base.cc:1034] running on GCE node
I20260619 11:05:20.568194 21462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:20.568768 21462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:20.569947 21462 hybrid_clock.cc:648] HybridClock initialized: now 1781867120569940 us; error 35 us; skew 500 ppm
I20260619 11:05:20.571583 21462 webserver.cc:460] Webserver started at http://127.16.61.2:34565/ using document root <none> and password file <none>
I20260619 11:05:20.572129 21462 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:20.572217 21462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:20.572433 21462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:20.574293 21462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "3d8d2f4b34a5483381d8c02b4a20b449"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "5c08a58974d09fef1dfbbfc520cc4a10"
server_key_iv: "c7df0c063d8ce8af3ab4f089f184eec6"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.574831 21462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "3d8d2f4b34a5483381d8c02b4a20b449"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "5c08a58974d09fef1dfbbfc520cc4a10"
server_key_iv: "c7df0c063d8ce8af3ab4f089f184eec6"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.578181 21462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:05:20.580493 21476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.581703 21462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:20.581841 21462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "3d8d2f4b34a5483381d8c02b4a20b449"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "5c08a58974d09fef1dfbbfc520cc4a10"
server_key_iv: "c7df0c063d8ce8af3ab4f089f184eec6"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.581949 21462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:20.586169 21462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:20.586707 21462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:20.586879 21462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:20.587368 21462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:20.588217 21462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:20.588281 21462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.588342 21462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:20.588397 21462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.596323 21462 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:41711
I20260619 11:05:20.596343 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:41711 every 8 connection(s)
I20260619 11:05:20.597105 21462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:20.602461 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.598732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:52791 (local address 127.16.61.62:40103)
0619 11:05:20.598872 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:20.598877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:20.599881 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 11:05:20.601054 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:20.601056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:20.601089 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:20.601160 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:20.602092 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:20.602094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:20.602096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:20.602106 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:20.602124 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:20.602131 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:20.602201 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:20.602326 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 11:05:20.602360 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:05:20.603232 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.599117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40103 (local address 127.16.61.2:52791)
0619 11:05:20.599752 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:20.599788 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:20.600888 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:20.601300 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:20.601306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:20.601388 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:20.601856 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:20.601858 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:20.601894 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:20.601956 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:20.602205 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:20.602211 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:20.602214 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:05:20.602347 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 11:05:20.602559 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260619 11:05:20.604286 21590 heartbeater.cc:347] Connected to a master server at 127.16.61.62:40103
I20260619 11:05:20.604466 21590 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:20.604770 21590 heartbeater.cc:511] Master 127.16.61.62:40103 requested a full tablet report, sending...
I20260619 11:05:20.605643 21271 ts_manager.cc:194] Registered new tserver with Master: 3d8d2f4b34a5483381d8c02b4a20b449 (127.16.61.2:41711)
I20260619 11:05:20.607084 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 21462
I20260619 11:05:20.607209 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:20.607455 16628 external_mini_cluster.cc:1428] Setting key 76228fa35efab5c537d195ef0ae6603a
I20260619 11:05:20.609066 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40103
--builtin_ntp_servers=127.16.61.20:34465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 11:05:20.711484 21593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:20.711726 21593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:20.711763 21593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:20.715548 21593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:20.715588 21593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:20.715639 21593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:20.720389 21593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:34465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:40103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:20.721275 21593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:20.722467 21593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:20.730974 21598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:20.731015 21599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:20.730934 21601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:20.731118 21593 server_base.cc:1034] running on GCE node
I20260619 11:05:20.731494 21593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:20.732064 21593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:20.733211 21593 hybrid_clock.cc:648] HybridClock initialized: now 1781867120733194 us; error 25 us; skew 500 ppm
I20260619 11:05:20.734907 21593 webserver.cc:460] Webserver started at http://127.16.61.3:37795/ using document root <none> and password file <none>
I20260619 11:05:20.735427 21593 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:20.735489 21593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:20.735774 21593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:20.737623 21593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "b9b73c9430e443b9a9f3a03b5b746ba3"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "56a5de1eae721e552ff4e405c831547f"
server_key_iv: "662ba38a0a01a1a99b37aee027416169"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.738155 21593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "b9b73c9430e443b9a9f3a03b5b746ba3"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "56a5de1eae721e552ff4e405c831547f"
server_key_iv: "662ba38a0a01a1a99b37aee027416169"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.741482 21593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:20.743647 21607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.744789 21593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:20.745010 21593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "b9b73c9430e443b9a9f3a03b5b746ba3"
format_stamp: "Formatted at 2026-06-19 11:05:20 on dist-test-slave-8vcz"
server_key: "56a5de1eae721e552ff4e405c831547f"
server_key_iv: "662ba38a0a01a1a99b37aee027416169"
server_key_version: "encryptionkey@0"
I20260619 11:05:20.745115 21593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:20.749503 21593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:20.750042 21593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:20.750234 21593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:20.750727 21593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:20.751536 21593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:20.751601 21593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.751682 21593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:20.751737 21593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:20.759653 21593 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:35679
I20260619 11:05:20.759719 21720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:35679 every 8 connection(s)
I20260619 11:05:20.760510 21593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:20.764201 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 21593
I20260619 11:05:20.764315 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:20.764616 16628 external_mini_cluster.cc:1428] Setting key 7c8ff4348458347f05dece2fe21b7e55
I20260619 11:05:20.766453 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.762107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:35193 (local address 127.16.61.62:40103)
0619 11:05:20.762240 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:20.762245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:20.763216 (+ 971us) server_negotiation.cc:407] Connection header received
0619 11:05:20.764654 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:20.764657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:20.764701 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:20.764778 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:20.765983 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:20.765987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:20.765989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:20.766001 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:20.766021 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:20.766030 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:20.766128 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:20.766285 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 11:05:20.766328 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:05:20.767308 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.762520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:40103 (local address 127.16.61.3:35193)
0619 11:05:20.763089 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:20.763124 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:20.764451 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:20.765011 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:20.765022 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:20.765134 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:20.765695 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:20.765699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:20.765750 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:20.765849 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:20.766126 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:20.766134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:20.766142 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 11:05:20.766313 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 11:05:20.766548 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260619 11:05:20.768364 21721 heartbeater.cc:347] Connected to a master server at 127.16.61.62:40103
I20260619 11:05:20.768599 21721 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:20.768985 21721 heartbeater.cc:511] Master 127.16.61.62:40103 requested a full tablet report, sending...
I20260619 11:05:20.769966 21271 ts_manager.cc:194] Registered new tserver with Master: b9b73c9430e443b9a9f3a03b5b746ba3 (127.16.61.3:35679)
I20260619 11:05:20.778472 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:05:20.781576 21310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.780282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.16.61.62:40103)
0619 11:05:20.780387 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:20.780391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:20.780548 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:05:20.780632 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:20.780634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:20.780661 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:20.780720 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:20.781287 (+ 567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:20.781289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:20.781291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:20.781300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:20.781314 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:20.781321 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:20.781385 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:20.781456 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 11:05:20.781482 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:05:20.783987 21271 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:20.786880 21271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:20.797549 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.795039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.16.61.3:35679)
0619 11:05:20.795308 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:20.795319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:20.795644 (+ 325us) server_negotiation.cc:407] Connection header received
0619 11:05:20.795725 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:20.795729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:20.795870 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:20.795978 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:20.796987 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:20.796993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:20.797004 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:20.797023 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:20.797056 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:20.797076 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:20.797182 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:20.797320 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 11:05:20.797394 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260619 11:05:20.797619 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.794801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:41711 (local address 127.0.0.1:52736)
0619 11:05:20.795504 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:20.795519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:20.795631 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:20.796060 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:20.796063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:20.796091 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:20.796842 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:20.796844 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:20.796854 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:20.796880 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:20.797247 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:20.797250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:20.797253 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:05:20.797324 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:05:20.797385 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260619 11:05:20.797845 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.794477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:35679 (local address 127.0.0.1:44768)
0619 11:05:20.795408 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:20.795454 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:20.795549 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:20.796011 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:20.796017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:20.796089 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:20.796792 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:20.796794 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:20.796809 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:20.796856 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:20.797167 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:20.797175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:20.797180 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:05:20.797306 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 11:05:20.797374 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"spinlock_wait_cycles":544128,"thread_start_us":74,"threads_started":1}
I20260619 11:05:20.798456 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.794358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:35099 (local address 127.0.0.1:49250)
0619 11:05:20.796319 (+ 1961us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:20.796333 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:20.796460 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:20.797441 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:20.797444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:20.797458 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:20.797848 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:20.797852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:20.797864 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:20.797890 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:20.798259 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:20.798262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:20.798265 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:05:20.798332 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 11:05:20.798361 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":73,"threads_started":1}
I20260619 11:05:20.798471 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.794964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.16.61.2:41711)
0619 11:05:20.795486 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:20.795496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:20.795619 (+ 123us) server_negotiation.cc:407] Connection header received
0619 11:05:20.795761 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:20.795766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:20.795907 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:20.796054 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:20.797007 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:20.797012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:20.797023 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:20.797034 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:20.797076 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:20.797109 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:20.797206 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:20.798248 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0619 11:05:20.798318 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":121,"threads_started":1}
I20260619 11:05:20.798870 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:20.794495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.16.61.1:35099)
0619 11:05:20.795375 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:20.795385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:20.796407 (+ 1022us) server_negotiation.cc:407] Connection header received
0619 11:05:20.796582 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:20.796589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:20.796746 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:20.796849 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:20.798030 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:20.798037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:20.798047 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:20.798058 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:20.798097 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:20.798128 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:20.798222 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:20.798671 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 11:05:20.798736 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":105,"threads_started":1}
I20260619 11:05:20.800737 21524 tablet_service.cc:1467] Processing CreateTablet for tablet 810cf34159e94fb79728e850cd70dc2b (DEFAULT_TABLE table=test-table [id=491d796d2c3047cf92b6b71c17a96f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:20.800782 21655 tablet_service.cc:1467] Processing CreateTablet for tablet 810cf34159e94fb79728e850cd70dc2b (DEFAULT_TABLE table=test-table [id=491d796d2c3047cf92b6b71c17a96f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:20.801214 21393 tablet_service.cc:1467] Processing CreateTablet for tablet 810cf34159e94fb79728e850cd70dc2b (DEFAULT_TABLE table=test-table [id=491d796d2c3047cf92b6b71c17a96f62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:20.801734 21524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810cf34159e94fb79728e850cd70dc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:20.801743 21655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810cf34159e94fb79728e850cd70dc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:20.802004 21393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810cf34159e94fb79728e850cd70dc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:20.807324 21738 tablet_bootstrap.cc:492] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Bootstrap starting.
I20260619 11:05:20.808606 21739 tablet_bootstrap.cc:492] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b: Bootstrap starting.
I20260619 11:05:20.810029 21738 tablet_bootstrap.cc:654] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:20.810606 21739 tablet_bootstrap.cc:654] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:20.810968 21738 log.cc:824] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:20.811363 21739 log.cc:824] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:20.811630 21740 tablet_bootstrap.cc:492] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Bootstrap starting.
I20260619 11:05:20.812769 21738 tablet_bootstrap.cc:492] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: No bootstrap required, opened a new log
I20260619 11:05:20.812971 21738 ts_tablet_manager.cc:1397] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:05:20.813177 21739 tablet_bootstrap.cc:492] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b: No bootstrap required, opened a new log
I20260619 11:05:20.813339 21739 ts_tablet_manager.cc:1397] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:05:20.813526 21740 tablet_bootstrap.cc:654] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:20.814261 21740 log.cc:824] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:20.816241 21740 tablet_bootstrap.cc:492] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: No bootstrap required, opened a new log
I20260619 11:05:20.816413 21740 ts_tablet_manager.cc:1397] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:05:20.817075 21738 raft_consensus.cc:348] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:20.817289 21738 raft_consensus.cc:374] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:20.817340 21738 raft_consensus.cc:729] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8d2f4b34a5483381d8c02b4a20b449, State: Initialized, Role: FOLLOWER
I20260619 11:05:20.817597 21739 raft_consensus.cc:348] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:20.817816 21739 raft_consensus.cc:374] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:20.817723 21738 consensus_queue.cc:260] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:20.817859 21739 raft_consensus.cc:729] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa8d04248b342b0821a4b06f374445b, State: Initialized, Role: FOLLOWER
I20260619 11:05:20.818254 21739 consensus_queue.cc:260] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:20.818637 21738 ts_tablet_manager.cc:1428] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:05:20.818967 21459 heartbeater.cc:503] Master 127.16.61.62:40103 was elected leader, sending a full tablet report...
I20260619 11:05:20.818969 21590 heartbeater.cc:503] Master 127.16.61.62:40103 was elected leader, sending a full tablet report...
I20260619 11:05:20.819161 21739 ts_tablet_manager.cc:1428] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260619 11:05:20.821213 21740 raft_consensus.cc:348] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:20.821467 21740 raft_consensus.cc:374] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:20.821532 21740 raft_consensus.cc:729] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b73c9430e443b9a9f3a03b5b746ba3, State: Initialized, Role: FOLLOWER
I20260619 11:05:20.821967 21740 consensus_queue.cc:260] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:20.822814 21740 ts_tablet_manager.cc:1428] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 11:05:20.823132 21721 heartbeater.cc:503] Master 127.16.61.62:40103 was elected leader, sending a full tablet report...
W20260619 11:05:20.847947 21591 tablet.cc:2369] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:20.934110 21460 tablet.cc:2369] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:21.011404 21722 tablet.cc:2369] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:21.182822 21744 raft_consensus.cc:482] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:21.183068 21744 raft_consensus.cc:504] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:21.184214 21744 leader_election.cc:288] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9aa8d04248b342b0821a4b06f374445b (127.16.61.1:35099), b9b73c9430e443b9a9f3a03b5b746ba3 (127.16.61.3:35679)
I20260619 11:05:21.186247 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.184594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:55191 (local address 127.16.61.1:35099)
0619 11:05:21.184780 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:21.184784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:21.184849 (+ 65us) server_negotiation.cc:407] Connection header received
0619 11:05:21.184986 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:21.184993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:21.185039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:21.185122 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:21.185873 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:21.185877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:21.185880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:21.185891 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:21.185917 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:21.185925 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:21.186020 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:21.186084 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 11:05:21.186124 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 11:05:21.186264 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.184437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:35099 (local address 127.16.61.2:55191)
0619 11:05:21.184745 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:21.184760 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:21.184863 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:21.185149 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:21.185152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:21.185168 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:21.185670 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:21.185672 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:21.185712 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:21.185746 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:21.186005 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:21.186008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:21.186011 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:05:21.186090 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:05:21.186124 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":68,"threads_started":1}
I20260619 11:05:21.186403 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.184576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:48245 (local address 127.16.61.3:35679)
0619 11:05:21.184714 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:21.184718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:21.185057 (+ 339us) server_negotiation.cc:407] Connection header received
0619 11:05:21.185154 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:21.185157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:21.185193 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:21.185282 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:21.186041 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:21.186045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:21.186047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:21.186058 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:21.186076 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:21.186091 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:21.186185 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:21.186257 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 11:05:21.186292 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:21.186430 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.184442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:35679 (local address 127.16.61.2:48245)
0619 11:05:21.184964 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:21.184977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:21.185058 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:21.185305 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:21.185307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:21.185319 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:21.185874 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:21.185877 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:21.185886 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:21.185916 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:21.186171 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:21.186174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:21.186177 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:05:21.186249 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:05:21.186284 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":12,"thread_start_us":75,"threads_started":1}
I20260619 11:05:21.186829 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810cf34159e94fb79728e850cd70dc2b" candidate_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa8d04248b342b0821a4b06f374445b" is_pre_election: true
I20260619 11:05:21.186918 21675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810cf34159e94fb79728e850cd70dc2b" candidate_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" is_pre_election: true
I20260619 11:05:21.187139 21675 raft_consensus.cc:2454] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d2f4b34a5483381d8c02b4a20b449 in term 0.
I20260619 11:05:21.187124 21413 raft_consensus.cc:2454] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d2f4b34a5483381d8c02b4a20b449 in term 0.
I20260619 11:05:21.187599 21480 leader_election.cc:302] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d8d2f4b34a5483381d8c02b4a20b449, 9aa8d04248b342b0821a4b06f374445b, b9b73c9430e443b9a9f3a03b5b746ba3; no voters:
I20260619 11:05:21.187878 21744 raft_consensus.cc:2781] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:21.187974 21744 raft_consensus.cc:482] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:21.188031 21744 raft_consensus.cc:3037] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:21.189020 21744 raft_consensus.cc:504] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:21.189338 21744 leader_election.cc:288] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [CANDIDATE]: Term 1 election: Requested vote from peers 9aa8d04248b342b0821a4b06f374445b (127.16.61.1:35099), b9b73c9430e443b9a9f3a03b5b746ba3 (127.16.61.3:35679)
I20260619 11:05:21.189678 21675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810cf34159e94fb79728e850cd70dc2b" candidate_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3"
I20260619 11:05:21.189678 21413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810cf34159e94fb79728e850cd70dc2b" candidate_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa8d04248b342b0821a4b06f374445b"
I20260619 11:05:21.189805 21413 raft_consensus.cc:3037] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:21.189805 21675 raft_consensus.cc:3037] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:21.191026 21675 raft_consensus.cc:2454] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d2f4b34a5483381d8c02b4a20b449 in term 1.
I20260619 11:05:21.191026 21413 raft_consensus.cc:2454] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d2f4b34a5483381d8c02b4a20b449 in term 1.
I20260619 11:05:21.191367 21477 leader_election.cc:302] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d2f4b34a5483381d8c02b4a20b449, b9b73c9430e443b9a9f3a03b5b746ba3; no voters:
I20260619 11:05:21.191557 21744 raft_consensus.cc:2781] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:21.191785 21744 raft_consensus.cc:686] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 1 LEADER]: Becoming Leader. State: Replica: 3d8d2f4b34a5483381d8c02b4a20b449, State: Running, Role: LEADER
I20260619 11:05:21.192101 21744 consensus_queue.cc:237] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } }
I20260619 11:05:21.194980 21271 catalog_manager.cc:5495] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d8d2f4b34a5483381d8c02b4a20b449 (127.16.61.2). New cstate: current_term: 1 leader_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d2f4b34a5483381d8c02b4a20b449" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 41711 } health_report { overall_health: HEALTHY } } }
I20260619 11:05:21.244683 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.243031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.16.61.2:41711)
0619 11:05:21.243183 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:21.243187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:21.243203 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:05:21.243370 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:21.243373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:21.243428 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:21.243522 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:21.244312 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:21.244315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:21.244317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:21.244327 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:21.244343 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:21.244350 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:21.244423 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:21.244490 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 11:05:21.244521 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 11:05:21.250396 21413 raft_consensus.cc:1261] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 1 FOLLOWER]: Refusing update from remote peer 3d8d2f4b34a5483381d8c02b4a20b449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:21.250396 21675 raft_consensus.cc:1261] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 1 FOLLOWER]: Refusing update from remote peer 3d8d2f4b34a5483381d8c02b4a20b449: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:21.251137 21749 consensus_queue.cc:1035] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b73c9430e443b9a9f3a03b5b746ba3" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:21.251309 21744 consensus_queue.cc:1035] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa8d04248b342b0821a4b06f374445b" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 35099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:21.258483 21753 mvcc.cc:205] Tried to move back new op lower bound from 7298527728634093568 to 7298527728405168128. Current Snapshot: MvccSnapshot[applied={T|T < 7298527728634093568}]
I20260619 11:05:21.265358 21270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:21.265532 21270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:21.267967 21270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f51f5b139c54e0cb7fc75bdf96924ec: Sending DeleteTablet for 3 replicas of tablet 810cf34159e94fb79728e850cd70dc2b
I20260619 11:05:21.268662 21393 tablet_service.cc:1514] Processing DeleteTablet for tablet 810cf34159e94fb79728e850cd70dc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:21 UTC) from {username='slave'} at 127.0.0.1:49250
I20260619 11:05:21.268760 21655 tablet_service.cc:1514] Processing DeleteTablet for tablet 810cf34159e94fb79728e850cd70dc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:21 UTC) from {username='slave'} at 127.0.0.1:44768
I20260619 11:05:21.268774 21524 tablet_service.cc:1514] Processing DeleteTablet for tablet 810cf34159e94fb79728e850cd70dc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:21 UTC) from {username='slave'} at 127.0.0.1:52736
I20260619 11:05:21.269122 21762 tablet_replica.cc:330] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b: stopping tablet replica
I20260619 11:05:21.269374 21761 tablet_replica.cc:330] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: stopping tablet replica
I20260619 11:05:21.269428 21762 raft_consensus.cc:2229] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:21.269649 21761 raft_consensus.cc:2229] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:21.269723 21762 raft_consensus.cc:2258] T 810cf34159e94fb79728e850cd70dc2b P 9aa8d04248b342b0821a4b06f374445b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:21.269956 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 21331
I20260619 11:05:21.270605 21763 tablet_replica.cc:330] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: stopping tablet replica
I20260619 11:05:21.270885 21763 raft_consensus.cc:2229] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:21.271209 21763 raft_consensus.cc:2258] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:21.272516 21761 raft_consensus.cc:2258] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:21.272652 21763 ts_tablet_manager.cc:1905] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:21.273854 21761 ts_tablet_manager.cc:1905] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:21.275349 21763 ts_tablet_manager.cc:1918] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:21.275516 21763 log.cc:1196] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/810cf34159e94fb79728e850cd70dc2b
I20260619 11:05:21.275936 21763 ts_tablet_manager.cc:1939] T 810cf34159e94fb79728e850cd70dc2b P 3d8d2f4b34a5483381d8c02b4a20b449: Deleting consensus metadata
I20260619 11:05:21.276556 21761 ts_tablet_manager.cc:1918] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:21.276702 21761 log.cc:1196] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/810cf34159e94fb79728e850cd70dc2b
I20260619 11:05:21.277170 21256 catalog_manager.cc:4841] TS 3d8d2f4b34a5483381d8c02b4a20b449 (127.16.61.2:41711): tablet 810cf34159e94fb79728e850cd70dc2b (table test-table [id=491d796d2c3047cf92b6b71c17a96f62]) successfully deleted
I20260619 11:05:21.277446 21761 ts_tablet_manager.cc:1939] T 810cf34159e94fb79728e850cd70dc2b P b9b73c9430e443b9a9f3a03b5b746ba3: Deleting consensus metadata
W20260619 11:05:21.277762 21258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:35099 (error 108)
I20260619 11:05:21.277968 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 21462
I20260619 11:05:21.279585 21255 catalog_manager.cc:4841] TS b9b73c9430e443b9a9f3a03b5b746ba3 (127.16.61.3:35679): tablet 810cf34159e94fb79728e850cd70dc2b (table test-table [id=491d796d2c3047cf92b6b71c17a96f62]) successfully deleted
I20260619 11:05:21.280133 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.279793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:35099 (local address 127.0.0.1:49260)
0619 11:05:21.279946 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:21.280032 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:35099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 11:05:21.280323 21258 catalog_manager.cc:4584] TS 9aa8d04248b342b0821a4b06f374445b (127.16.61.1:35099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 810cf34159e94fb79728e850cd70dc2b: Network error: Client connection negotiation failed: client connection to 127.16.61.1:35099: connect: Connection refused (error 111)
I20260619 11:05:21.283773 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 21593
I20260619 11:05:21.289516 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 21240
2026-06-19T11:05:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1244 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 11:05:21.310218 16628 mini_oidc.cc:152] Starting JWKS server
I20260619 11:05:21.311815 16628 webserver.cc:460] Webserver started at https://127.0.0.1:38941/ using document root <none> and password file <none>
I20260619 11:05:21.312000 16628 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:05:21.312275 16628 webserver.cc:460] Webserver started at http://127.0.0.1:38897/ using document root <none> and password file <none>
2026-06-19T11:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:21Z Disabled control of system clock
I20260619 11:05:21.320873 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:45431
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:42971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:45431
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38941/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/testchainca.cert with env {}
W20260619 11:05:21.425555 21770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:21.425789 21770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:21.425827 21770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:21.429689 21770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:21.429734 21770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:21.429750 21770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:21.429773 21770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:21.429788 21770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:05:21.429803 21770 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:05:21.429816 21770 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38941/jwks.json
I20260619 11:05:21.435309 21770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:42971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:45431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:45431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38941/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:21.436203 21770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:21.437050 21770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:21.444976 21778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:21.445017 21775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:21.445026 21776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:21.445420 21770 server_base.cc:1034] running on GCE node
I20260619 11:05:21.445935 21770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:21.446981 21770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:21.448163 21770 hybrid_clock.cc:648] HybridClock initialized: now 1781867121448140 us; error 34 us; skew 500 ppm
I20260619 11:05:21.449983 21770 webserver.cc:460] Webserver started at http://127.16.61.62:45457/ using document root <none> and password file <none>
I20260619 11:05:21.450531 21770 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:21.450591 21770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:21.450791 21770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:21.452777 21770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "8a6817873e9f4a658ad13d2a19b39e6e"
format_stamp: "Formatted at 2026-06-19 11:05:21 on dist-test-slave-8vcz"
server_key: "9c418954e477c9c64526d535f84bdab2"
server_key_iv: "c243fbb36e91d3852a7f082fb0af5e57"
server_key_version: "encryptionkey@0"
I20260619 11:05:21.453322 21770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "8a6817873e9f4a658ad13d2a19b39e6e"
format_stamp: "Formatted at 2026-06-19 11:05:21 on dist-test-slave-8vcz"
server_key: "9c418954e477c9c64526d535f84bdab2"
server_key_iv: "c243fbb36e91d3852a7f082fb0af5e57"
server_key_version: "encryptionkey@0"
I20260619 11:05:21.456746 21770 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 11:05:21.458935 21784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:21.460337 21770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:21.460489 21770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "8a6817873e9f4a658ad13d2a19b39e6e"
format_stamp: "Formatted at 2026-06-19 11:05:21 on dist-test-slave-8vcz"
server_key: "9c418954e477c9c64526d535f84bdab2"
server_key_iv: "c243fbb36e91d3852a7f082fb0af5e57"
server_key_version: "encryptionkey@0"
I20260619 11:05:21.460605 21770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:21.474398 21770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:21.475004 21770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:21.475200 21770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:21.482031 21836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:45431 every 8 connection(s)
I20260619 11:05:21.482029 21770 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:45431
I20260619 11:05:21.483026 21770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:21.485630 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 21770
I20260619 11:05:21.485762 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:21.486050 16628 external_mini_cluster.cc:1428] Setting key b66ba37ece5de3ec6f0cff1fd261f098
I20260619 11:05:21.486308 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:21.492820 21837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e: Bootstrap starting.
I20260619 11:05:21.494992 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.487312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36592 (local address 127.16.61.62:45431)
0619 11:05:21.487954 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:21.487969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:21.488004 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:05:21.488916 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:21.488933 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:21.489306 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:21.489566 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:21.489899 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.490813 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.492032 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.492475 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.492596 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:21.493402 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:21.493406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:21.493419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:21.493766 (+ 347us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:21.493818 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:21.493836 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:21.493918 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:21.494074 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 11:05:21.494342 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":143,"threads_started":1}
I20260619 11:05:21.495128 21837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:21.495894 21837 log.cc:824] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:21.497709 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:05:21.497768 21837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e: No bootstrap required, opened a new log
I20260619 11:05:21.501206 21837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6817873e9f4a658ad13d2a19b39e6e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45431 } }
I20260619 11:05:21.501396 21837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:21.501435 21837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6817873e9f4a658ad13d2a19b39e6e, State: Initialized, Role: FOLLOWER
I20260619 11:05:21.501752 21837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6817873e9f4a658ad13d2a19b39e6e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45431 } }
I20260619 11:05:21.501852 21837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:21.501893 21837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:21.501968 21837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:21.502673 21837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6817873e9f4a658ad13d2a19b39e6e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45431 } }
I20260619 11:05:21.502943 21837 leader_election.cc:302] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a6817873e9f4a658ad13d2a19b39e6e; no voters:
I20260619 11:05:21.503159 21837 leader_election.cc:288] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:21.503289 21842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:21.503536 21842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [term 1 LEADER]: Becoming Leader. State: Replica: 8a6817873e9f4a658ad13d2a19b39e6e, State: Running, Role: LEADER
I20260619 11:05:21.503866 21842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6817873e9f4a658ad13d2a19b39e6e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45431 } }
I20260619 11:05:21.504284 21837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:21.505556 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a6817873e9f4a658ad13d2a19b39e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6817873e9f4a658ad13d2a19b39e6e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45431 } } }
I20260619 11:05:21.505556 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a6817873e9f4a658ad13d2a19b39e6e. Latest consensus state: current_term: 1 leader_uuid: "8a6817873e9f4a658ad13d2a19b39e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6817873e9f4a658ad13d2a19b39e6e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45431 } } }
I20260619 11:05:21.505697 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:21.505697 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e [sys.catalog]: This master's current role is: LEADER
W20260619 11:05:21.508814 21857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:05:21.508879 21857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:05:21.508962 21852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:21.509596 21852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:21.514216 21852 catalog_manager.cc:1347] Generated new cluster ID: d97fab2c85574e80a09deab49d7ab938
I20260619 11:05:21.514276 21852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:21.526942 21852 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:21.527900 21852 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:21.535110 21852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a6817873e9f4a658ad13d2a19b39e6e: Generated new TSK 0
I20260619 11:05:21.535841 21852 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 11:05:21.568322 21867 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 11:05:21.568445 21867 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 11:05:21.568912 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.547468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.16.61.62:45431)
0619 11:05:21.547585 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:21.547592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:21.547799 (+ 207us) server_negotiation.cc:407] Connection header received
0619 11:05:21.548000 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:21.548002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:21.548046 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:21.548127 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:05:21.548396 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.548902 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.550116 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.550324 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.550396 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:21.550735 (+ 339us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:05:21.568609 (+ 17874us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 11:05:21.568813 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36606: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 11:05:21.568992 21840 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36606: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 11:05:21.570317 21866 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:21.547356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45431 (local address 127.0.0.1:36606)
0619 11:05:21.547715 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:21.547731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:21.547850 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:21.548165 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:21.548169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:21.548181 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:05:21.548294 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:21.548300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.549006 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.549010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:21.550008 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:21.550018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.550427 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.550431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:21.550517 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:21.550566 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:05:21.568815 (+ 18249us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 11:05:21.569076 (+ 261us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.61.62:45431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.61.62:45431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38941/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 11:05:21.571038 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 21770
2026-06-19T11:05:21Z chronyd exiting
I20260619 11:05:21.608024 16628 test_util.cc:175] -----------------------------------------------
I20260619 11:05:21.608112 16628 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781867094678158-16628-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 11:05:21.609125 16628 mini_oidc.cc:152] Starting JWKS server
I20260619 11:05:21.609935 16628 webserver.cc:460] Webserver started at https://127.0.0.1:44967/ using document root <none> and password file <none>
I20260619 11:05:21.610056 16628 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:05:21.610364 16628 webserver.cc:460] Webserver started at http://127.0.0.1:39119/ using document root <none> and password file <none>
2026-06-19T11:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:21Z Disabled control of system clock
I20260619 11:05:21.618357 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44873
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:36487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:44873
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44967/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/testchainca.cert with env {}
W20260619 11:05:21.721065 21873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:21.721307 21873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:21.721345 21873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:21.725265 21873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:21.725309 21873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:21.725324 21873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:21.725342 21873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:21.725397 21873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:05:21.725423 21873 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:05:21.725454 21873 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44967/jwks.json
I20260619 11:05:21.730557 21873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:44873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44967/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:21.731485 21873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:21.732407 21873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:21.740144 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:21.740233 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:21.740150 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:21.740396 21873 server_base.cc:1034] running on GCE node
I20260619 11:05:21.740931 21873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:21.741916 21873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:21.743072 21873 hybrid_clock.cc:648] HybridClock initialized: now 1781867121743057 us; error 24 us; skew 500 ppm
I20260619 11:05:21.744840 21873 webserver.cc:460] Webserver started at http://127.16.61.62:37443/ using document root <none> and password file <none>
I20260619 11:05:21.745373 21873 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:21.745443 21873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:21.745675 21873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:21.747525 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "ca9432c29c5248d993f855153a03cedb"
format_stamp: "Formatted at 2026-06-19 11:05:21 on dist-test-slave-8vcz"
server_key: "caed31e082e46473d79db71e0ee69d5e"
server_key_iv: "2efa2a1af76b0a40606596d6fee14904"
server_key_version: "encryptionkey@0"
I20260619 11:05:21.748147 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "ca9432c29c5248d993f855153a03cedb"
format_stamp: "Formatted at 2026-06-19 11:05:21 on dist-test-slave-8vcz"
server_key: "caed31e082e46473d79db71e0ee69d5e"
server_key_iv: "2efa2a1af76b0a40606596d6fee14904"
server_key_version: "encryptionkey@0"
I20260619 11:05:21.751397 21873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:21.753628 21887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:21.755046 21873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:05:21.755177 21873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "ca9432c29c5248d993f855153a03cedb"
format_stamp: "Formatted at 2026-06-19 11:05:21 on dist-test-slave-8vcz"
server_key: "caed31e082e46473d79db71e0ee69d5e"
server_key_iv: "2efa2a1af76b0a40606596d6fee14904"
server_key_version: "encryptionkey@0"
I20260619 11:05:21.755300 21873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:21.780723 21873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:21.781373 21873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:21.781567 21873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:21.788276 21873 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:44873
I20260619 11:05:21.788286 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:44873 every 8 connection(s)
I20260619 11:05:21.789191 21873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:21.792132 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:21.793119 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 21873
I20260619 11:05:21.793210 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:21.793500 16628 external_mini_cluster.cc:1428] Setting key e0c71bcaa8ce4e59fdb79d3424ccb774
I20260619 11:05:21.798197 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb: Bootstrap starting.
I20260619 11:05:21.800575 21940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:21.801436 21940 log.cc:824] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:21.803319 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.794740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59544 (local address 127.16.61.62:44873)
0619 11:05:21.795354 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:21.795374 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:21.795414 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:05:21.796238 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:21.796261 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:21.796717 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:21.797069 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:21.797477 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.798866 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.799819 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.800467 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.800632 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:21.801439 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:21.801447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:21.801465 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:21.801927 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:21.801975 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:21.801994 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:21.802068 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:21.802251 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 11:05:21.802487 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
I20260619 11:05:21.803444 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb: No bootstrap required, opened a new log
I20260619 11:05:21.805876 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:05:21.807312 21940 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9432c29c5248d993f855153a03cedb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44873 } }
I20260619 11:05:21.807576 21940 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:21.807657 21940 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9432c29c5248d993f855153a03cedb, State: Initialized, Role: FOLLOWER
I20260619 11:05:21.808048 21940 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9432c29c5248d993f855153a03cedb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44873 } }
I20260619 11:05:21.808189 21940 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:21.808280 21940 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:21.808374 21940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:21.809384 21940 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9432c29c5248d993f855153a03cedb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44873 } }
I20260619 11:05:21.809700 21940 leader_election.cc:302] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca9432c29c5248d993f855153a03cedb; no voters:
I20260619 11:05:21.809984 21940 leader_election.cc:288] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:21.810117 21945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:21.810424 21945 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [term 1 LEADER]: Becoming Leader. State: Replica: ca9432c29c5248d993f855153a03cedb, State: Running, Role: LEADER
I20260619 11:05:21.810727 21945 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9432c29c5248d993f855153a03cedb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44873 } }
I20260619 11:05:21.811151 21940 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:21.812433 21947 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9432c29c5248d993f855153a03cedb. Latest consensus state: current_term: 1 leader_uuid: "ca9432c29c5248d993f855153a03cedb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9432c29c5248d993f855153a03cedb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44873 } } }
I20260619 11:05:21.812558 21947 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:21.812417 21946 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca9432c29c5248d993f855153a03cedb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca9432c29c5248d993f855153a03cedb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44873 } } }
I20260619 11:05:21.812712 21946 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:21.812953 21955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:21.816094 21955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:21.821099 21955 catalog_manager.cc:1347] Generated new cluster ID: 17b7cff903074ab4a583b03c2ee66ad1
I20260619 11:05:21.821159 21955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:21.830063 21955 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:21.830897 21955 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:21.837679 21955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca9432c29c5248d993f855153a03cedb: Generated new TSK 0
I20260619 11:05:21.838258 21955 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 11:05:21.868834 21970 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 11:05:21.868916 21970 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 11:05:21.869468 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:21.854064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.16.61.62:44873)
0619 11:05:21.854197 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:21.854203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:21.854377 (+ 174us) server_negotiation.cc:407] Connection header received
0619 11:05:21.854531 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:21.854533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:21.854576 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:21.854654 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:05:21.854898 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.855426 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.856647 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.856859 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.856941 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:21.857244 (+ 303us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:05:21.869179 (+ 11935us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 11:05:21.869368 (+ 189us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59556: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 11:05:21.869544 21943 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59556: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 11:05:21.869671 21969 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:21.853990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44873 (local address 127.0.0.1:59556)
0619 11:05:21.854288 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:21.854303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:21.854410 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:21.854676 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:21.854679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:21.854686 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:05:21.854791 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:21.854797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.855527 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.855530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:21.856522 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:21.856532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:21.856979 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:21.856982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:21.857070 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:21.857096 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:05:21.869343 (+ 12247us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 11:05:21.869499 (+ 156us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.61.62:44873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44967/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":95,"threads_started":1}
I20260619 11:05:21.870326 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 21873
2026-06-19T11:05:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (299 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 11:05:21.909108 16628 mini_oidc.cc:152] Starting JWKS server
I20260619 11:05:21.909942 16628 webserver.cc:460] Webserver started at https://127.0.0.1:37333/ using document root <none> and password file <none>
I20260619 11:05:21.910041 16628 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 11:05:21.910287 16628 webserver.cc:460] Webserver started at http://127.0.0.1:42287/ using document root <none> and password file <none>
2026-06-19T11:05:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:21Z Disabled control of system clock
I20260619 11:05:21.918687 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:36379
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:36379
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37333/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 11:05:22.022922 21976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:22.023195 21976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:22.023269 21976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:22.027130 21976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:22.027211 21976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:22.027266 21976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:22.027329 21976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:22.027369 21976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:05:22.027405 21976 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:05:22.027441 21976 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37333/jwks.json
I20260619 11:05:22.032028 21976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:36379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37333/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:22.033017 21976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:22.034325 21976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:22.042342 21981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:22.042400 21982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:22.042634 21976 server_base.cc:1034] running on GCE node
W20260619 11:05:22.042416 21984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:22.043311 21976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:22.044399 21976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:22.045581 21976 hybrid_clock.cc:648] HybridClock initialized: now 1781867122045564 us; error 30 us; skew 500 ppm
I20260619 11:05:22.051859 21976 webserver.cc:460] Webserver started at http://127.16.61.62:38209/ using document root <none> and password file <none>
I20260619 11:05:22.052410 21976 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:22.052501 21976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:22.052726 21976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:22.054656 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "3088760e4f1846c28e075556ffa9bcf9"
format_stamp: "Formatted at 2026-06-19 11:05:22 on dist-test-slave-8vcz"
server_key: "804114ce975a82b85e9674b5bd762d5a"
server_key_iv: "36d9e21840c7fc0795c96543fb1eecaf"
server_key_version: "encryptionkey@0"
I20260619 11:05:22.055195 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "3088760e4f1846c28e075556ffa9bcf9"
format_stamp: "Formatted at 2026-06-19 11:05:22 on dist-test-slave-8vcz"
server_key: "804114ce975a82b85e9674b5bd762d5a"
server_key_iv: "36d9e21840c7fc0795c96543fb1eecaf"
server_key_version: "encryptionkey@0"
I20260619 11:05:22.058622 21976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:05:22.060860 21990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:22.061983 21976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:22.062111 21976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "3088760e4f1846c28e075556ffa9bcf9"
format_stamp: "Formatted at 2026-06-19 11:05:22 on dist-test-slave-8vcz"
server_key: "804114ce975a82b85e9674b5bd762d5a"
server_key_iv: "36d9e21840c7fc0795c96543fb1eecaf"
server_key_version: "encryptionkey@0"
I20260619 11:05:22.062215 21976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:22.071198 21976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:22.071776 21976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:22.071962 21976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:22.078372 21976 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:36379
I20260619 11:05:22.078378 22042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:36379 every 8 connection(s)
I20260619 11:05:22.079208 21976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:22.081984 22043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:22.083514 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 21976
I20260619 11:05:22.083628 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:22.083930 16628 external_mini_cluster.cc:1428] Setting key aa6b3ee4bd70a89274bc5e9f975c0770
I20260619 11:05:22.087939 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9: Bootstrap starting.
I20260619 11:05:22.089954 22043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:22.090626 22043 log.cc:824] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:22.092427 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9: No bootstrap required, opened a new log
I20260619 11:05:22.092432 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:22.085235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.16.61.62:36379)
0619 11:05:22.085805 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:22.085818 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:22.085849 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:05:22.086453 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:22.086468 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:22.086812 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:22.087125 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:22.087533 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.088574 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.089414 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.089871 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.090049 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:22.090841 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:22.090846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:22.090856 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:22.091202 (+ 346us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:22.091251 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:22.091269 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:22.091375 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:22.091503 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 11:05:22.091797 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
I20260619 11:05:22.095016 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:05:22.096297 22043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3088760e4f1846c28e075556ffa9bcf9" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36379 } }
I20260619 11:05:22.096580 22043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:22.096658 22043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3088760e4f1846c28e075556ffa9bcf9, State: Initialized, Role: FOLLOWER
I20260619 11:05:22.097079 22043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3088760e4f1846c28e075556ffa9bcf9" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36379 } }
I20260619 11:05:22.097223 22043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:22.097289 22043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:22.097393 22043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:22.098192 22043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3088760e4f1846c28e075556ffa9bcf9" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36379 } }
I20260619 11:05:22.098507 22043 leader_election.cc:302] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3088760e4f1846c28e075556ffa9bcf9; no voters:
I20260619 11:05:22.098784 22043 leader_election.cc:288] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:22.098865 22048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:22.099128 22048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3088760e4f1846c28e075556ffa9bcf9, State: Running, Role: LEADER
I20260619 11:05:22.099439 22048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3088760e4f1846c28e075556ffa9bcf9" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36379 } }
I20260619 11:05:22.099957 22043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:22.101007 22050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3088760e4f1846c28e075556ffa9bcf9. Latest consensus state: current_term: 1 leader_uuid: "3088760e4f1846c28e075556ffa9bcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3088760e4f1846c28e075556ffa9bcf9" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36379 } } }
I20260619 11:05:22.101141 22050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:22.101428 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3088760e4f1846c28e075556ffa9bcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3088760e4f1846c28e075556ffa9bcf9" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36379 } } }
I20260619 11:05:22.101547 22057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:22.102023 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:22.104465 22057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:22.109493 22057 catalog_manager.cc:1347] Generated new cluster ID: f782f2029758449e8e5db234303b9f99
I20260619 11:05:22.109606 22057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:22.140117 22057 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:22.141021 22057 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:22.147938 22057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3088760e4f1846c28e075556ffa9bcf9: Generated new TSK 0
I20260619 11:05:22.148669 22057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:05:22.171574 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:22.165965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.16.61.62:36379)
0619 11:05:22.166106 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:22.166111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:22.166330 (+ 219us) server_negotiation.cc:407] Connection header received
0619 11:05:22.166552 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:22.166556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:22.166621 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:22.166711 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:05:22.166956 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.167475 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.168723 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.169067 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.169140 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:22.169559 (+ 419us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:05:22.171186 (+ 1627us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 11:05:22.171441 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50414: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 11:05:22.171648 22072 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:22.165898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:36379 (local address 127.0.0.1:50414)
0619 11:05:22.166251 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:22.166266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:22.166378 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:22.166731 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:22.166735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:22.166743 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:05:22.166848 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:22.166854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.167597 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.167600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:22.168587 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:22.168597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.169257 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.169261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:22.169344 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:22.169367 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:05:22.171354 (+ 1987us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 11:05:22.171488 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.61.62:36379: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
W20260619 11:05:22.172454 22046 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50414: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37333/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260619 11:05:22.175420 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 21976
E20260619 11:05:22.185045 22073 webserver.cc:533] Webserver: sslize failed:
E20260619 11:05:22.185123 22073 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T11:05:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T11:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:22Z Disabled control of system clock
I20260619 11:05:22.222855 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:38613
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:34015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:38613
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/testchainca.cert with env {}
W20260619 11:05:22.326663 22077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:22.326925 22077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:22.326992 22077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:22.330940 22077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:22.331020 22077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:22.331050 22077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:22.331087 22077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:22.331128 22077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 11:05:22.331161 22077 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 11:05:22.331194 22077 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 11:05:22.336234 22077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:34015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:38613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:38613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:22.337224 22077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:22.338109 22077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:22.345880 22082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:22.345865 22085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:22.345849 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:22.345908 22077 server_base.cc:1034] running on GCE node
I20260619 11:05:22.346681 22077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:22.347740 22077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:22.348913 22077 hybrid_clock.cc:648] HybridClock initialized: now 1781867122348897 us; error 27 us; skew 500 ppm
I20260619 11:05:22.350515 22077 webserver.cc:460] Webserver started at http://127.16.61.62:43005/ using document root <none> and password file <none>
I20260619 11:05:22.351039 22077 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:22.351127 22077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:22.351315 22077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:22.353206 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "b6e2adb4c292465c912e722c3cabef64"
format_stamp: "Formatted at 2026-06-19 11:05:22 on dist-test-slave-8vcz"
server_key: "7557a6a12d1af0d561df9133c1064a2e"
server_key_iv: "d70896cbead03c81a255457b154cb574"
server_key_version: "encryptionkey@0"
I20260619 11:05:22.353749 22077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "b6e2adb4c292465c912e722c3cabef64"
format_stamp: "Formatted at 2026-06-19 11:05:22 on dist-test-slave-8vcz"
server_key: "7557a6a12d1af0d561df9133c1064a2e"
server_key_iv: "d70896cbead03c81a255457b154cb574"
server_key_version: "encryptionkey@0"
I20260619 11:05:22.357039 22077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:22.359241 22091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:22.360368 22077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:22.360507 22077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "b6e2adb4c292465c912e722c3cabef64"
format_stamp: "Formatted at 2026-06-19 11:05:22 on dist-test-slave-8vcz"
server_key: "7557a6a12d1af0d561df9133c1064a2e"
server_key_iv: "d70896cbead03c81a255457b154cb574"
server_key_version: "encryptionkey@0"
I20260619 11:05:22.360610 22077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:22.416854 22077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:22.417536 22077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:22.417733 22077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:22.424212 22077 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:38613
I20260619 11:05:22.424222 22143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:38613 every 8 connection(s)
I20260619 11:05:22.425078 22077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:22.427704 22144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:22.432628 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 22077
I20260619 11:05:22.432744 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:22.432816 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64: Bootstrap starting.
I20260619 11:05:22.433032 16628 external_mini_cluster.cc:1428] Setting key 5f7d8c8b0730daff4bf5bb19eb2c6004
I20260619 11:05:22.435220 22144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:22.435968 22144 log.cc:824] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:22.437780 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64: No bootstrap required, opened a new log
I20260619 11:05:22.441648 22144 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e2adb4c292465c912e722c3cabef64" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38613 } }
I20260619 11:05:22.441833 22144 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:22.441865 22144 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e2adb4c292465c912e722c3cabef64, State: Initialized, Role: FOLLOWER
I20260619 11:05:22.441893 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:22.434267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.16.61.62:38613)
0619 11:05:22.434832 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:22.434845 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:22.434876 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:05:22.435526 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:22.435542 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:22.435918 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:22.436269 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:22.436665 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.437767 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.438941 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.439362 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.439482 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:22.440266 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:22.440271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:22.440281 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:22.440621 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:22.440668 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:22.440686 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:22.440757 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:22.440950 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 11:05:22.441210 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260619 11:05:22.442337 22144 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e2adb4c292465c912e722c3cabef64" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38613 } }
I20260619 11:05:22.442453 22144 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:22.442534 22144 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:22.442637 22144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:22.443373 22144 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e2adb4c292465c912e722c3cabef64" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38613 } }
I20260619 11:05:22.443766 22144 leader_election.cc:302] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6e2adb4c292465c912e722c3cabef64; no voters:
I20260619 11:05:22.444084 22144 leader_election.cc:288] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:22.444181 22149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:22.444370 22149 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [term 1 LEADER]: Becoming Leader. State: Replica: b6e2adb4c292465c912e722c3cabef64, State: Running, Role: LEADER
I20260619 11:05:22.444545 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:05:22.444631 22149 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e2adb4c292465c912e722c3cabef64" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38613 } }
I20260619 11:05:22.445178 22144 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:22.445488 22149 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6e2adb4c292465c912e722c3cabef64. Latest consensus state: current_term: 1 leader_uuid: "b6e2adb4c292465c912e722c3cabef64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e2adb4c292465c912e722c3cabef64" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38613 } } }
I20260619 11:05:22.445597 22149 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:22.445924 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6e2adb4c292465c912e722c3cabef64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e2adb4c292465c912e722c3cabef64" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38613 } } }
I20260619 11:05:22.446034 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:22.446170 22153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:22.449036 22153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:22.453927 22153 catalog_manager.cc:1347] Generated new cluster ID: 5ee3f38e0b094376ab5bc4631da7fd43
I20260619 11:05:22.453994 22153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:22.460053 22153 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:22.460815 22153 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:22.468075 22153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6e2adb4c292465c912e722c3cabef64: Generated new TSK 0
I20260619 11:05:22.468611 22153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:05:22.567852 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:22.478971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.16.61.62:38613)
0619 11:05:22.479090 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:22.479095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:22.479230 (+ 135us) server_negotiation.cc:407] Connection header received
0619 11:05:22.479336 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:22.479339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:22.479389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:22.479503 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0619 11:05:22.479713 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.480509 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.481314 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.481663 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.481763 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:22.482009 (+ 246us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 11:05:22.567379 (+ 85370us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 11:05:22.567700 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 11:05:22.567926 22147 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51364: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 11:05:22.568279 22173 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:22.478876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:38613 (local address 127.0.0.1:51364)
0619 11:05:22.479150 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:22.479162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:22.479246 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:22.479505 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:22.479507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:22.479513 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 11:05:22.479591 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:22.479594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.480624 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.480626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:22.481209 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:22.481215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:22.481775 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:22.481778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:22.481837 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:22.481856 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 11:05:22.567596 (+ 85740us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 11:05:22.567830 (+ 234us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.61.62:38613: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260619 11:05:22.569111 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 22077
2026-06-19T11:05:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (373 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (114 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (115 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:22 dist-test-slave-8vcz krb5kdc[22185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:22 dist-test-slave-8vcz krb5kdc[22185](info): set up 2 sockets
Jun 19 11:05:22 dist-test-slave-8vcz krb5kdc[22185](info): commencing operation
krb5kdc: starting...
W20260619 11:05:23.824023 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:23 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:23.936743 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44815
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:44815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:24.041168 22198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:24.041427 22198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:24.041510 22198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:24.045487 22198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:24.045562 22198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:24.045599 22198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:24.045655 22198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:24.045694 22198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:24.050704 22198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:44815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:24.051640 22198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:24.052623 22198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:24.060539 22206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:24.060580 22203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:24.060539 22204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:24.060819 22198 server_base.cc:1034] running on GCE node
I20260619 11:05:24.061357 22198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:24.062278 22198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:24.063442 22198 hybrid_clock.cc:648] HybridClock initialized: now 1781867124063426 us; error 34 us; skew 500 ppm
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.066251 22198 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:24.067030 22198 webserver.cc:460] Webserver started at http://127.16.61.62:42497/ using document root <none> and password file <none>
I20260619 11:05:24.067554 22198 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:24.067615 22198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:24.067870 22198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:24.069761 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "21b8513fbecf4da5a41b4e49585eb387"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "454641f31b3b955b419f445bae67a6ce"
server_key_iv: "bba64badaf8a7ce82b6d36a35d686725"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.070288 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "21b8513fbecf4da5a41b4e49585eb387"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "454641f31b3b955b419f445bae67a6ce"
server_key_iv: "bba64badaf8a7ce82b6d36a35d686725"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.073709 22198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 11:05:24.075903 22213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.077020 22198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 11:05:24.077153 22198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "21b8513fbecf4da5a41b4e49585eb387"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "454641f31b3b955b419f445bae67a6ce"
server_key_iv: "bba64badaf8a7ce82b6d36a35d686725"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.077255 22198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:24.086380 22198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:24.089382 22198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:24.089569 22198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:24.096390 22198 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:44815
I20260619 11:05:24.096416 22265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:44815 every 8 connection(s)
I20260619 11:05:24.097244 22198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:24.099756 22266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:24.101758 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 22198
I20260619 11:05:24.101876 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:24.102156 16628 external_mini_cluster.cc:1428] Setting key 6f6c6bd93111bf716bb56e71844d8ce4
I20260619 11:05:24.105449 22266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387: Bootstrap starting.
I20260619 11:05:24.107486 22266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:24.108239 22266 log.cc:824] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:24.110356 22266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387: No bootstrap required, opened a new log
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:24.114286 22266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b8513fbecf4da5a41b4e49585eb387" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44815 } }
I20260619 11:05:24.114565 22266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:24.114646 22266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b8513fbecf4da5a41b4e49585eb387, State: Initialized, Role: FOLLOWER
I20260619 11:05:24.115036 22266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b8513fbecf4da5a41b4e49585eb387" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44815 } }
I20260619 11:05:24.115180 22266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:24.115264 22266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:24.115362 22266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:24.116281 22266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b8513fbecf4da5a41b4e49585eb387" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44815 } }
I20260619 11:05:24.116596 22266 leader_election.cc:302] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21b8513fbecf4da5a41b4e49585eb387; no voters:
I20260619 11:05:24.116871 22266 leader_election.cc:288] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:24.116961 22271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:24.116979 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.103489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.16.61.62:44815)
0619 11:05:24.103899 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.103912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.103946 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:05:24.104716 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.104733 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.105033 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.105381 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.106139 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.107095 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.108032 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.108552 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.108725 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.111403 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:24.111409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:24.111425 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:24.111457 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:24.113818 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.114229 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.114243 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.114248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.114383 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.114691 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.114700 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.114701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.115142 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:24.115391 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:24.115901 (+ 510us) server_negotiation.cc:299] Negotiation successful
0619 11:05:24.116147 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":140,"threads_started":1}
I20260619 11:05:24.117190 22271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [term 1 LEADER]: Becoming Leader. State: Replica: 21b8513fbecf4da5a41b4e49585eb387, State: Running, Role: LEADER
I20260619 11:05:24.117511 22271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b8513fbecf4da5a41b4e49585eb387" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44815 } }
I20260619 11:05:24.118005 22266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:24.118645 22272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21b8513fbecf4da5a41b4e49585eb387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b8513fbecf4da5a41b4e49585eb387" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44815 } } }
I20260619 11:05:24.118755 22272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:24.119004 22273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21b8513fbecf4da5a41b4e49585eb387. Latest consensus state: current_term: 1 leader_uuid: "21b8513fbecf4da5a41b4e49585eb387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b8513fbecf4da5a41b4e49585eb387" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44815 } } }
I20260619 11:05:24.119107 22273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:24.119374 22278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:24.122327 22278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:24.127877 22278 catalog_manager.cc:1347] Generated new cluster ID: d6888b24b110420189364aa5a616df0c
I20260619 11:05:24.127930 22278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:24.147699 22278 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:24.148547 22278 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:24.153390 22278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21b8513fbecf4da5a41b4e49585eb387: Generated new TSK 0
I20260619 11:05:24.153990 22278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:24.188318 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44815
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:24.305480 22292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:24.305696 22292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:24.305735 22292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:24.309618 22292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:24.309664 22292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:24.309717 22292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:24.314635 22292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:24.315524 22292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:24.316473 22292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:24.325384 22300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:24.325433 22298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:24.325383 22297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:24.325465 22292 server_base.cc:1034] running on GCE node
I20260619 11:05:24.325984 22292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:24.326614 22292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:24.327800 22292 hybrid_clock.cc:648] HybridClock initialized: now 1781867124327771 us; error 41 us; skew 500 ppm
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.330196 22292 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:24.330997 22292 webserver.cc:460] Webserver started at http://127.16.61.1:35617/ using document root <none> and password file <none>
I20260619 11:05:24.331504 22292 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:24.331591 22292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:24.331849 22292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:24.333750 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "bafab3aedddf494fac9a33b3b8b98a66"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "cebace483fa8c0038838270a1bdec629"
server_key_iv: "9be35a0641d729195a9904920b4c8a7d"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.334255 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "bafab3aedddf494fac9a33b3b8b98a66"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "cebace483fa8c0038838270a1bdec629"
server_key_iv: "9be35a0641d729195a9904920b4c8a7d"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.337623 22292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:24.339761 22307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.340817 22292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:24.340940 22292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "bafab3aedddf494fac9a33b3b8b98a66"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "cebace483fa8c0038838270a1bdec629"
server_key_iv: "9be35a0641d729195a9904920b4c8a7d"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.341038 22292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:24.361619 22292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:24.364177 22292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:24.364367 22292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:24.364882 22292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:24.365628 22292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:24.365702 22292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.365772 22292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:24.365808 22292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.375090 22292 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:46281
I20260619 11:05:24.375186 22420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:46281 every 8 connection(s)
I20260619 11:05:24.376053 22292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:24.384446 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 22292
I20260619 11:05:24.384577 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:24.384914 16628 external_mini_cluster.cc:1428] Setting key e490e4621582ea29a2120d2031f4ec03
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:24.391218 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.379424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:34611 (local address 127.16.61.62:44815)
0619 11:05:24.379558 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.379562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.379585 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:05:24.380762 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.380765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.380811 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.380887 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.381872 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.382397 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.383419 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.383644 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.383801 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.386902 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:24.386910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:24.386914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:24.386942 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:24.389051 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.389691 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.389698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.389702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.389794 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.390120 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.390126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.390131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.390375 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:24.390475 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:24.390924 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 11:05:24.391053 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 11:05:24.392030 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.378391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44815 (local address 127.16.61.1:34611)
0619 11:05:24.379296 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:24.379350 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:24.380585 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:24.381007 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:24.381013 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:24.381326 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:24.381747 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:24.381759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.382499 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.382501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:24.383320 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:24.383328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.383841 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.383844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:24.383974 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.384494 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:24.384517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:24.386703 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:24.389164 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:24.389171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:24.389191 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:24.389582 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:24.389899 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:24.389901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:24.389902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:24.390029 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:24.390480 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:24.390485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:24.390694 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 11:05:24.390880 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 11:05:24.391180 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":147,"threads_started":1}
I20260619 11:05:24.393149 22421 heartbeater.cc:347] Connected to a master server at 127.16.61.62:44815
I20260619 11:05:24.393396 22421 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:24.393959 22421 heartbeater.cc:511] Master 127.16.61.62:44815 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:05:24.395457 22230 ts_manager.cc:194] Registered new tserver with Master: bafab3aedddf494fac9a33b3b8b98a66 (127.16.61.1:46281)
I20260619 11:05:24.396624 22230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:34611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:24.412197 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44815
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:24.514895 22426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:24.515111 22426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:24.515151 22426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:24.519166 22426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:24.519217 22426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:24.519275 22426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:24.524219 22426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:24.525020 22426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:24.525843 22426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:24.534456 22432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:24.534413 22434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:24.534415 22431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:24.534916 22426 server_base.cc:1034] running on GCE node
I20260619 11:05:24.535231 22426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:24.535782 22426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:24.536959 22426 hybrid_clock.cc:648] HybridClock initialized: now 1781867124536929 us; error 40 us; skew 500 ppm
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.539520 22426 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:24.540362 22426 webserver.cc:460] Webserver started at http://127.16.61.2:44521/ using document root <none> and password file <none>
I20260619 11:05:24.540864 22426 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:24.540946 22426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:24.541134 22426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:24.542970 22426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "c06555626c2e47f2876641b47797fad8"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "2a36d23f663636e5f19403f9d2b4cb41"
server_key_iv: "a8f50705282b234b7536bcb648f579e3"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.543488 22426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "c06555626c2e47f2876641b47797fad8"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "2a36d23f663636e5f19403f9d2b4cb41"
server_key_iv: "a8f50705282b234b7536bcb648f579e3"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.546834 22426 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 11:05:24.549053 22441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.550132 22426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:24.550256 22426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "c06555626c2e47f2876641b47797fad8"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "2a36d23f663636e5f19403f9d2b4cb41"
server_key_iv: "a8f50705282b234b7536bcb648f579e3"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.550380 22426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:24.571264 22426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:24.573851 22426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:24.574033 22426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:24.574553 22426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:24.575277 22426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:24.575351 22426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.575431 22426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:24.575472 22426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.583770 22426 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:46569
I20260619 11:05:24.583799 22554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:46569 every 8 connection(s)
I20260619 11:05:24.584568 22426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:24.587369 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 22426
I20260619 11:05:24.587508 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:24.587895 16628 external_mini_cluster.cc:1428] Setting key 001cf8154c1c1ccfdbbe29d3f89ee16b
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
I20260619 11:05:24.598650 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.586215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:55771 (local address 127.16.61.62:44815)
0619 11:05:24.586344 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.586348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.587055 (+ 707us) server_negotiation.cc:407] Connection header received
0619 11:05:24.588847 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.588851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.588904 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.589002 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.590456 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.591102 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.592306 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.592546 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.592858 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.595222 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:24.595225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:24.595228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:24.595257 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:24.596694 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.597207 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.597212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.597214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.597294 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.597627 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.597631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.597632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.597811 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:24.597918 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:24.598371 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 11:05:24.598494 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:05:24.599793 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.586514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44815 (local address 127.16.61.2:55771)
0619 11:05:24.586912 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:24.586951 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:24.588633 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:24.589168 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:24.589174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:24.589633 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:24.590294 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:24.590316 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.591236 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.591238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:24.592181 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:24.592190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.592879 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.592883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:24.593064 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.593710 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:24.593726 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:24.595067 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:24.596817 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:24.596822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:24.596835 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:24.597101 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:24.597393 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:24.597395 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:24.597397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:24.597533 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:24.597914 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:24.597920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:24.598150 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 11:05:24.598350 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 11:05:24.598672 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":116,"threads_started":1}
I20260619 11:05:24.600950 22555 heartbeater.cc:347] Connected to a master server at 127.16.61.62:44815
I20260619 11:05:24.601202 22555 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:24.601743 22555 heartbeater.cc:511] Master 127.16.61.62:44815 requested a full tablet report, sending...
I20260619 11:05:24.602857 22230 ts_manager.cc:194] Registered new tserver with Master: c06555626c2e47f2876641b47797fad8 (127.16.61.2:46569)
I20260619 11:05:24.603390 22230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:55771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:05:24.614346 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44815
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:41281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:24.717139 22560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:24.717356 22560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:24.717396 22560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:24.721514 22560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:24.721561 22560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:24.721616 22560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:24.726398 22560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:24.727185 22560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:24.728048 22560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:24.736515 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:24.736564 22565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:24.736614 22560 server_base.cc:1034] running on GCE node
W20260619 11:05:24.736799 22568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:24.737125 22560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:24.737661 22560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:24.738831 22560 hybrid_clock.cc:648] HybridClock initialized: now 1781867124738825 us; error 27 us; skew 500 ppm
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.741384 22560 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:24.742206 22560 webserver.cc:460] Webserver started at http://127.16.61.3:33317/ using document root <none> and password file <none>
I20260619 11:05:24.742712 22560 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:24.742791 22560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:24.742980 22560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:24.744872 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "f5959e4dabb54126b55bcf4500c82cd9"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "5ff97b9517e78ef0d4b2985f46d8b291"
server_key_iv: "96a66dbaf34b48beecbca04c449ab68c"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.745399 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "f5959e4dabb54126b55bcf4500c82cd9"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "5ff97b9517e78ef0d4b2985f46d8b291"
server_key_iv: "96a66dbaf34b48beecbca04c449ab68c"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.748683 22560 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 11:05:24.750823 22575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.751947 22560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:24.752061 22560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "f5959e4dabb54126b55bcf4500c82cd9"
format_stamp: "Formatted at 2026-06-19 11:05:24 on dist-test-slave-8vcz"
server_key: "5ff97b9517e78ef0d4b2985f46d8b291"
server_key_iv: "96a66dbaf34b48beecbca04c449ab68c"
server_key_version: "encryptionkey@0"
I20260619 11:05:24.752156 22560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:24.761767 22560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:24.764606 22560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:24.764784 22560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:24.765268 22560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:24.765967 22560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:24.766032 22560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.766096 22560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:24.766136 22560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:24.774461 22560 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:44659
I20260619 11:05:24.774487 22688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:44659 every 8 connection(s)
I20260619 11:05:24.775213 22560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:24.779629 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 22560
I20260619 11:05:24.779765 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:24.780083 16628 external_mini_cluster.cc:1428] Setting key 75d351bf3dcda4dafe98b2756cf298bb
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:24.787760 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.776820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:34829 (local address 127.16.61.62:44815)
0619 11:05:24.776947 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.776951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.777664 (+ 713us) server_negotiation.cc:407] Connection header received
0619 11:05:24.778920 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.778923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.778987 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.779067 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.780435 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.781232 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.782475 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.782681 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.782761 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.784952 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:24.784955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:24.784957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:24.784980 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:24.786129 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.786550 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.786553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.786554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.786621 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.786882 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.786884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.786885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.787032 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:24.787103 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:24.787507 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 11:05:24.787591 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:05:24.788404 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.777150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44815 (local address 127.16.61.3:34829)
0619 11:05:24.777538 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:24.777574 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:24.778720 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:24.779228 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:24.779234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:24.779645 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:24.780261 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:24.780280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.781388 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.781391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:24.782359 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:24.782367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.782795 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.782798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:24.782922 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.783387 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:24.783402 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:24.784815 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:24.786225 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:24.786229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:24.786239 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:24.786470 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:24.786695 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:24.786697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:24.786698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:24.786810 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:24.787112 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:24.787116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:24.787312 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 11:05:24.787454 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 11:05:24.787659 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":109,"threads_started":1}
I20260619 11:05:24.789429 22689 heartbeater.cc:347] Connected to a master server at 127.16.61.62:44815
I20260619 11:05:24.789651 22689 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:24.790077 22689 heartbeater.cc:511] Master 127.16.61.62:44815 requested a full tablet report, sending...
I20260619 11:05:24.790966 22230 ts_manager.cc:194] Registered new tserver with Master: f5959e4dabb54126b55bcf4500c82cd9 (127.16.61.3:44659)
I20260619 11:05:24.791469 22230 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:34829
I20260619 11:05:24.793311 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/test-admin.keytab.
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.807401 16628 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 11:05:24.809976 22696 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:24.808614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.16.61.1:46281)
0619 11:05:24.808968 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.808977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.808996 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:05:24.809038 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.809042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.809185 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.809322 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.809769 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49638: BlockingRecv error: recv got EOF from 127.0.0.1:49638 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
W20260619 11:05:24.810012 16628 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5cc' of type FILE
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.810917 16628 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.811794 16628 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:05:24.820815 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.812530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.16.61.1:46281)
0619 11:05:24.812650 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.812655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.812750 (+ 95us) server_negotiation.cc:407] Connection header received
0619 11:05:24.812880 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.812883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.812929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.813013 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.813494 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.814231 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.815024 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.815364 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.815475 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.817582 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:24.817597 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:24.817607 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:24.817627 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:24.819066 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.819549 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.819554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.819555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.819628 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.819943 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.819952 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.819954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.820162 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:24.820377 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:24.820557 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 11:05:24.820696 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 11:05:24.824072 22696 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:24.823164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.16.61.1:46281)
0619 11:05:24.823276 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.823280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.823353 (+ 73us) server_negotiation.cc:407] Connection header received
0619 11:05:24.823479 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.823482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.823523 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.823602 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.823980 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49664: BlockingRecv error: recv got EOF from 127.0.0.1:49664 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 11:05:24.824226 16628 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5cc' of type FILE
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.825139 16628 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.825963 16628 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:05:24.834046 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.826689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.16.61.1:46281)
0619 11:05:24.826801 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.826805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.826904 (+ 99us) server_negotiation.cc:407] Connection header received
0619 11:05:24.827038 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.827040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.827082 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.827162 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.827633 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.828169 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.828929 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.829157 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.829268 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.831183 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:24.831189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:24.831196 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:24.831216 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:24.832560 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.832957 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.832963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.832964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.833047 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.833307 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.833312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.833313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.833462 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:24.833589 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:24.833811 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 11:05:24.833919 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 11:05:24.836292 22696 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:24.835407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.16.61.1:46281)
0619 11:05:24.835493 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.835497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.835605 (+ 108us) server_negotiation.cc:407] Connection header received
0619 11:05:24.835750 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.835752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.835788 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.835850 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.836187 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49688: BlockingRecv error: recv got EOF from 127.0.0.1:49688 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 11:05:24.836436 16628 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781867094678158-16628-0/krb5kdc/krb5cc' of type FILE
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.837306 16628 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:24.838138 16628 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:05:24.846253 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:24.838828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.16.61.1:46281)
0619 11:05:24.838950 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:24.838954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:24.839041 (+ 87us) server_negotiation.cc:407] Connection header received
0619 11:05:24.839168 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:24.839171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:24.839213 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:24.839309 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:24.839807 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.840268 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.841020 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:24.841225 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:24.841330 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:24.843364 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:24.843366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:24.843368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:24.843386 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:24.844766 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.845151 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.845155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.845157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.845232 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:24.845507 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:24.845510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:24.845512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:24.845671 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:24.845791 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:24.846014 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 11:05:24.846114 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:05:24.847067 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 22292
I20260619 11:05:24.852960 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 22426
I20260619 11:05:24.858191 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 22560
I20260619 11:05:24.863062 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 22198
2026-06-19T11:05:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2065 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22709](info): set up 2 sockets
Jun 19 11:05:24 dist-test-slave-8vcz krb5kdc[22709](info): commencing operation
krb5kdc: starting...
W20260619 11:05:25.896457 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:25 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:25Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.62.keytab.
Entry for principal oryx/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.62.keytab.
I20260619 11:05:26.014365 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:45373
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.62.keytab
--principal=oryx/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:45373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:26.116935 22722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:26.117190 22722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:26.117260 22722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:26.121735 22722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:26.121816 22722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:26.121863 22722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:26.121898 22722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:26.121935 22722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:26.127245 22722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:45373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.62.keytab
--principal=oryx/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:26.128310 22722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:26.129281 22722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:26.137272 22730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:26.137272 22727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:26.137310 22728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:26.137413 22722 server_base.cc:1034] running on GCE node
I20260619 11:05:26.138156 22722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:26.139241 22722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:26.140409 22722 hybrid_clock.cc:648] HybridClock initialized: now 1781867126140392 us; error 30 us; skew 500 ppm
Jun 19 11:05:26 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:26.143044 22722 init.cc:377] Logged in from keytab as oryx/127.16.61.62@KRBTEST.COM (short username oryx)
I20260619 11:05:26.143957 22722 webserver.cc:460] Webserver started at http://127.16.61.62:39013/ using document root <none> and password file <none>
I20260619 11:05:26.144505 22722 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:26.144598 22722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:26.144822 22722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:26.146872 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "9ca84261526b4b5da2e43f353fd49c8f"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "e9332ec772ba82e7130af0090a132ef3"
server_key_iv: "c150fb5fe6ffb4697f2d43a272ab42fc"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.147439 22722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "9ca84261526b4b5da2e43f353fd49c8f"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "e9332ec772ba82e7130af0090a132ef3"
server_key_iv: "c150fb5fe6ffb4697f2d43a272ab42fc"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.151046 22722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:05:26.153323 22737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.154448 22722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:26.154583 22722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "9ca84261526b4b5da2e43f353fd49c8f"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "e9332ec772ba82e7130af0090a132ef3"
server_key_iv: "c150fb5fe6ffb4697f2d43a272ab42fc"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.154687 22722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:26.169243 22722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:26.171833 22722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:26.172012 22722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:26.178447 22722 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:45373
I20260619 11:05:26.178553 22789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:45373 every 8 connection(s)
I20260619 11:05:26.179422 22722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:26.181999 22790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:26.187165 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: Bootstrap starting.
I20260619 11:05:26.189440 22790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:26.189541 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 22722
I20260619 11:05:26.189697 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:26.189965 16628 external_mini_cluster.cc:1428] Setting key c31904ed5890a8cd3920da23203904d9
I20260619 11:05:26.190253 22790 log.cc:824] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:26.192175 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: No bootstrap required, opened a new log
I20260619 11:05:26.196393 22790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } }
I20260619 11:05:26.196614 22790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:26.196667 22790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca84261526b4b5da2e43f353fd49c8f, State: Initialized, Role: FOLLOWER
I20260619 11:05:26.197058 22790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } }
I20260619 11:05:26.197202 22790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:26.197260 22790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:26.197395 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:26.198168 22790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } }
I20260619 11:05:26.198484 22790 leader_election.cc:302] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ca84261526b4b5da2e43f353fd49c8f; no voters:
Jun 19 11:05:26 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:26.198766 22790 leader_election.cc:288] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:26.198870 22795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:26.199079 22795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 1 LEADER]: Becoming Leader. State: Replica: 9ca84261526b4b5da2e43f353fd49c8f, State: Running, Role: LEADER
I20260619 11:05:26.199381 22795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } }
I20260619 11:05:26.200033 22790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:26.201206 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } }
I20260619 11:05:26.201331 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:26.201627 22797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca84261526b4b5da2e43f353fd49c8f. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } }
I20260619 11:05:26.201713 22797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:26.202656 22804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:26.205075 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:26.191042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.16.61.62:45373)
0619 11:05:26.191450 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:26.191469 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:26.191513 (+ 44us) server_negotiation.cc:407] Connection header received
0619 11:05:26.192173 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:26.192189 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:26.192539 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:26.192802 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:26.193668 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.194726 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.195989 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.196449 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.196618 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:26.199428 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:26.199435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:26.199449 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:26.199630 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:26.202005 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:26.202557 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:26.202563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:26.202569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:26.202721 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:26.203007 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:26.203010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:26.203012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:26.203466 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:26.203686 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:26.203936 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 11:05:26.204215 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":136,"threads_started":1}
I20260619 11:05:26.205809 22804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:26.210427 22804 catalog_manager.cc:1347] Generated new cluster ID: ed7a528273cd4b0e842c3ac9f472b358
I20260619 11:05:26.210492 22804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:26.220459 22804 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:26.221318 22804 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:26.225847 22804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: Generated new TSK 0
I20260619 11:05:26.226495 22804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab.
Entry for principal oryx/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab.
I20260619 11:05:26.255416 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:45373
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab
--principal=oryx/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:26.362782 22816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:26.363009 22816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:26.363049 22816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:26.366835 22816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:26.366881 22816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:26.366935 22816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:26.372048 22816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab
--principal=oryx/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:26.373255 22816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:26.374545 22816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:26.385205 22821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:26.385205 22824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:26.385218 22822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:26.385289 22816 server_base.cc:1034] running on GCE node
I20260619 11:05:26.385769 22816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:26.386412 22816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:26.387598 22816 hybrid_clock.cc:648] HybridClock initialized: now 1781867126387587 us; error 26 us; skew 500 ppm
Jun 19 11:05:26 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:26.390201 22816 init.cc:377] Logged in from keytab as oryx/127.16.61.1@KRBTEST.COM (short username oryx)
I20260619 11:05:26.391033 22816 webserver.cc:460] Webserver started at http://127.16.61.1:34871/ using document root <none> and password file <none>
I20260619 11:05:26.391551 22816 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:26.391639 22816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:26.391892 22816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:26.394315 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "069869d675a740a09d0f034a0566ddd9"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "b3bc1fa38d2f918270dffb5c4fcfa1d4"
server_key_iv: "65817f9ec40ae8d98294290c5b7ae3ca"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.394966 22816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "069869d675a740a09d0f034a0566ddd9"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "b3bc1fa38d2f918270dffb5c4fcfa1d4"
server_key_iv: "65817f9ec40ae8d98294290c5b7ae3ca"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.398516 22816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:05:26.400974 22831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.402237 22816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:26.402376 22816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "069869d675a740a09d0f034a0566ddd9"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "b3bc1fa38d2f918270dffb5c4fcfa1d4"
server_key_iv: "65817f9ec40ae8d98294290c5b7ae3ca"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.402482 22816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:26.416980 22816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:26.419567 22816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:26.419790 22816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:26.420307 22816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:26.421099 22816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:26.421160 22816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.421219 22816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:26.421267 22816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.429908 22816 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:42999
I20260619 11:05:26.429931 22944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:42999 every 8 connection(s)
I20260619 11:05:26.430698 22816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:26.431332 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 22816
I20260619 11:05:26.431444 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:26.431778 16628 external_mini_cluster.cc:1428] Setting key 99963589a705bba85af5d17665e58bfe
Jun 19 11:05:26 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.2@KRBTEST.COM" created.
I20260619 11:05:26.444967 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:26.432483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:47765 (local address 127.16.61.62:45373)
0619 11:05:26.432600 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:26.432603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:26.433567 (+ 964us) server_negotiation.cc:407] Connection header received
0619 11:05:26.434806 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:26.434810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:26.434856 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:26.434945 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:26.436198 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.436717 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.438563 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.438787 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.438862 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:26.441422 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:26.441430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:26.441432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:26.441453 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:26.442881 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:26.443394 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:26.443396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:26.443398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:26.443468 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:26.443821 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:26.443825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:26.443828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:26.444065 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:26.444168 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:26.444762 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 11:05:26.444853 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:05:26.446041 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:26.432949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.1:47765)
0619 11:05:26.433442 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:26.433480 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:26.434623 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:26.435100 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:26.435106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:26.435431 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:26.436047 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:26.436068 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.436875 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.436882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:26.438432 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:26.438447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.438943 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.438948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:26.439149 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:26.439731 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:26.439751 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:26.441267 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:26.443003 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:26.443008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:26.443018 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:26.443312 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:26.443548 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:26.443550 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:26.443551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:26.443725 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:26.444182 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:26.444190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:26.444507 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 11:05:26.444707 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 11:05:26.445038 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":153,"threads_started":1}
I20260619 11:05:26.447253 22945 heartbeater.cc:347] Connected to a master server at 127.16.61.62:45373
I20260619 11:05:26.447515 22945 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:26.448006 22945 heartbeater.cc:511] Master 127.16.61.62:45373 requested a full tablet report, sending...
I20260619 11:05:26.449467 22754 ts_manager.cc:194] Registered new tserver with Master: 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999)
I20260619 11:05:26.450519 22754 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:47765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab.
Entry for principal oryx/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab.
I20260619 11:05:26.459211 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:45373
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab
--principal=oryx/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:26.563133 22950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:26.563432 22950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:26.563478 22950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:26.567315 22950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:26.567365 22950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:26.567418 22950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:26.572242 22950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab
--principal=oryx/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:26.573099 22950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:26.574023 22950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:05:26.582805 22950 server_base.cc:1034] running on GCE node
W20260619 11:05:26.582938 22955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:26.582870 22958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:26.582824 22956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:26.583441 22950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:26.584080 22950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:26.585233 22950 hybrid_clock.cc:648] HybridClock initialized: now 1781867126585187 us; error 58 us; skew 500 ppm
Jun 19 11:05:26 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:26.587822 22950 init.cc:377] Logged in from keytab as oryx/127.16.61.2@KRBTEST.COM (short username oryx)
I20260619 11:05:26.588537 22950 webserver.cc:460] Webserver started at http://127.16.61.2:40641/ using document root <none> and password file <none>
I20260619 11:05:26.589008 22950 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:26.589061 22950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:26.589208 22950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:26.591020 22950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "c6cd46533d1e402fb5661105f73741d6"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "4da5f03da00dc8782b3428995d0b2d83"
server_key_iv: "6686b4e371c119c4d2ccc276212dd197"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.591550 22950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "c6cd46533d1e402fb5661105f73741d6"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "4da5f03da00dc8782b3428995d0b2d83"
server_key_iv: "6686b4e371c119c4d2ccc276212dd197"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.594861 22950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:05:26.597060 22965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.598297 22950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:26.598423 22950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "c6cd46533d1e402fb5661105f73741d6"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "4da5f03da00dc8782b3428995d0b2d83"
server_key_iv: "6686b4e371c119c4d2ccc276212dd197"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.598523 22950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:26.610021 22950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:26.612567 22950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:26.612751 22950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:26.613283 22950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:26.614012 22950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:26.614055 22950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.614123 22950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:26.614163 22950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.622509 22950 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:36265
I20260619 11:05:26.622535 23078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:36265 every 8 connection(s)
I20260619 11:05:26.623323 22950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:26.624316 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 22950
I20260619 11:05:26.624433 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:26.624706 16628 external_mini_cluster.cc:1428] Setting key 678fda178a27e252011e02b3772107a9
Jun 19 11:05:26 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.3@KRBTEST.COM" created.
I20260619 11:05:26.637439 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:26.625157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:41445 (local address 127.16.61.62:45373)
0619 11:05:26.625294 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:26.625298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:26.626172 (+ 874us) server_negotiation.cc:407] Connection header received
0619 11:05:26.627519 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:26.627523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:26.627576 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:26.627699 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:26.628787 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.629658 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.630850 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.631303 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.631441 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:26.633965 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:26.633969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:26.633972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:26.634001 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:26.635294 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:26.635795 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:26.635801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:26.635802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:26.635910 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:26.636291 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:26.636294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:26.636295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:26.636502 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:26.636636 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:26.637148 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 11:05:26.637265 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:05:26.638185 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:26.625561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.2:41445)
0619 11:05:26.626021 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:26.626068 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:26.627310 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:26.627874 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:26.627880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:26.628204 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:26.628648 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:26.628661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.629825 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.629829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:26.630723 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:26.630731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.631450 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.631453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:26.631585 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:26.632214 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:26.632230 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:26.633802 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:26.635396 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:26.635400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:26.635410 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:26.635634 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:26.636009 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:26.636012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:26.636013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:26.636140 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:26.636639 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:26.636647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:26.636911 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 11:05:26.637088 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 11:05:26.637408 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":133,"threads_started":1}
I20260619 11:05:26.639532 23079 heartbeater.cc:347] Connected to a master server at 127.16.61.62:45373
I20260619 11:05:26.639850 23079 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:26.640343 23079 heartbeater.cc:511] Master 127.16.61.62:45373 requested a full tablet report, sending...
I20260619 11:05:26.641495 22754 ts_manager.cc:194] Registered new tserver with Master: c6cd46533d1e402fb5661105f73741d6 (127.16.61.2:36265)
I20260619 11:05:26.642380 22754 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:41445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab.
Entry for principal oryx/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab.
I20260619 11:05:26.652711 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:45373
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab
--principal=oryx/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:26.759006 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:26.759322 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:26.759410 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:26.763266 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:26.763352 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:26.763440 23084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:26.768410 23084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab
--principal=oryx/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:45373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:26.769320 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:26.770242 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:26.779110 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:26.779106 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:26.779083 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:26.779558 23084 server_base.cc:1034] running on GCE node
I20260619 11:05:26.779965 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:26.780545 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:26.781718 23084 hybrid_clock.cc:648] HybridClock initialized: now 1781867126781701 us; error 30 us; skew 500 ppm
Jun 19 11:05:26 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:26.784266 23084 init.cc:377] Logged in from keytab as oryx/127.16.61.3@KRBTEST.COM (short username oryx)
I20260619 11:05:26.785087 23084 webserver.cc:460] Webserver started at http://127.16.61.3:33027/ using document root <none> and password file <none>
I20260619 11:05:26.785601 23084 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:26.785691 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:26.785918 23084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:26.787868 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "25b2f6b36d284fb7aab35e486f3b2015"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "abe6ea2449dc5df098ce9f5dc231e86a"
server_key_iv: "8f57e1ca3d9124e0c908c5532f53675f"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.788398 23084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "25b2f6b36d284fb7aab35e486f3b2015"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "abe6ea2449dc5df098ce9f5dc231e86a"
server_key_iv: "8f57e1ca3d9124e0c908c5532f53675f"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.791852 23084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:26.794001 23099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.795099 23084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:26.795228 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "25b2f6b36d284fb7aab35e486f3b2015"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "abe6ea2449dc5df098ce9f5dc231e86a"
server_key_iv: "8f57e1ca3d9124e0c908c5532f53675f"
server_key_version: "encryptionkey@0"
I20260619 11:05:26.795333 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:26.819737 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:26.822546 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:26.822739 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:26.823282 23084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:26.824101 23084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:26.824173 23084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.824239 23084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:26.824281 23084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:26.832302 23084 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:32829
I20260619 11:05:26.832381 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:32829 every 8 connection(s)
I20260619 11:05:26.833150 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:26.838119 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 23084
I20260619 11:05:26.838243 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:26.838491 16628 external_mini_cluster.cc:1428] Setting key 81ccc00e63f677dab2e4b577e81bc240
Jun 19 11:05:26 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:26.845852 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:26.834966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:58545 (local address 127.16.61.62:45373)
0619 11:05:26.835093 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:26.835097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:26.835927 (+ 830us) server_negotiation.cc:407] Connection header received
0619 11:05:26.837257 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:26.837260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:26.837315 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:26.837385 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:26.838513 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.839066 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.840181 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.840401 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.840506 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:26.842714 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:26.842717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:26.842722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:26.842747 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:26.843992 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:26.844461 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:26.844465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:26.844467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:26.844540 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:26.844872 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:26.844876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:26.844878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:26.845039 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:26.845138 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:26.845584 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 11:05:26.845685 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:26.846724 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:26.835321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.3:58545)
0619 11:05:26.835796 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:26.835833 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:26.837048 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:26.837541 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:26.837548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:26.837871 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:26.838376 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:26.838391 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.839187 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.839191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:26.840058 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:26.840067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:26.840533 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:26.840537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:26.840699 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:26.841239 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:26.841254 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:26.842565 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:26.844099 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:26.844105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:26.844118 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:26.844361 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:26.844632 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:26.844637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:26.844639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:26.844778 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:26.845148 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:26.845156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:26.845372 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 11:05:26.845545 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 11:05:26.845837 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":129,"threads_started":1}
I20260619 11:05:26.847733 23213 heartbeater.cc:347] Connected to a master server at 127.16.61.62:45373
I20260619 11:05:26.847952 23213 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:26.848403 23213 heartbeater.cc:511] Master 127.16.61.62:45373 requested a full tablet report, sending...
I20260619 11:05:26.849300 22754 ts_manager.cc:194] Registered new tserver with Master: 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829)
I20260619 11:05:26.849782 22754 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:58545
I20260619 11:05:26.851923 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.61.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.61.keytab.
Entry for principal oryx/127.16.61.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.61.keytab.
I20260619 11:05:26.876291 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.61:44913
--webserver_interface=127.16.61.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.61.keytab
--principal=oryx/127.16.61.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:45373,127.16.61.61:44913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:26.981223 23218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:26.981468 23218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:26.981545 23218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:26.985718 23218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:26.985791 23218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:26.985836 23218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:26.985867 23218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:26.985920 23218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:26.990921 23218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:45373,127.16.61.61:44913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.61.keytab
--principal=oryx/127.16.61.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.61:44913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.61.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
W20260619 11:05:26.991138 23218 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.61.62:45373,127.16.61.61:44913'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 11:05:26.991969 23218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:26.992880 23218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:27.000558 23226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.000630 23223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.000591 23224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:27.000801 23218 server_base.cc:1034] running on GCE node
I20260619 11:05:27.001343 23218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:27.002461 23218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:27.003623 23218 hybrid_clock.cc:648] HybridClock initialized: now 1781867127003605 us; error 29 us; skew 500 ppm
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:27.006212 23218 init.cc:377] Logged in from keytab as oryx/127.16.61.61@KRBTEST.COM (short username oryx)
I20260619 11:05:27.007046 23218 webserver.cc:460] Webserver started at http://127.16.61.61:32779/ using document root <none> and password file <none>
I20260619 11:05:27.007570 23218 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:27.007630 23218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:27.007845 23218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:27.009697 23218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data/instance:
uuid: "19087d9aa3d849c1bad2e98b83a56c72"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "2777c9fb579dd44cd45c5bc6d49d53fb"
server_key_iv: "c3d7ee91129475cabdd6564bf349d1ed"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.010226 23218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal/instance:
uuid: "19087d9aa3d849c1bad2e98b83a56c72"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "2777c9fb579dd44cd45c5bc6d49d53fb"
server_key_iv: "c3d7ee91129475cabdd6564bf349d1ed"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.013825 23218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:27.016064 23233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.017231 23218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:27.017366 23218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
uuid: "19087d9aa3d849c1bad2e98b83a56c72"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "2777c9fb579dd44cd45c5bc6d49d53fb"
server_key_iv: "c3d7ee91129475cabdd6564bf349d1ed"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.017469 23218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:27.074540 23218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:27.077276 23218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:27.077481 23218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:27.084151 23218 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.61:44913
I20260619 11:05:27.084450 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.61:44913 every 8 connection(s)
I20260619 11:05:27.085088 23218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
I20260619 11:05:27.087634 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:27.091115 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:05:27.091979 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 23218
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.61@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.61.61@KRBTEST.COM
I20260619 11:05:27.104576 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.092904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.16.61.62:45373)
0619 11:05:27.093056 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.093060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.094014 (+ 954us) server_negotiation.cc:407] Connection header received
0619 11:05:27.094765 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.094767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.094810 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.094881 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.095764 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.096267 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.097357 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.097647 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.097723 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.100631 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.100636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.100641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.100674 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.102282 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.102829 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.102831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.102833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.102904 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.103262 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.103265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.103267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.103577 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.103649 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.104231 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.104392 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 11:05:27.105518 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.093294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.16.61.61:44913)
0619 11:05:27.093750 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.093770 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.093806 (+ 36us) server_negotiation.cc:407] Connection header received
0619 11:05:27.094567 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.094588 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.094806 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.094986 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.095657 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.096700 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.097535 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.097949 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.098042 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.101038 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.101045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.101057 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.101084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.103094 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.103548 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.103552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.103553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.103642 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.103930 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.103933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.103935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.104187 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.104358 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.104574 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.104703 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":311,"thread_start_us":132,"threads_started":1}
I20260619 11:05:27.105518 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.093294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:55476)
0619 11:05:27.093867 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.093909 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.094567 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.094951 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.094956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.095358 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.095581 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.095620 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.096399 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.096403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.097243 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.097251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.097772 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.097776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.097994 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.098777 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.098791 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.100421 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.102385 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.102395 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.102404 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.102737 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.102991 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.102997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.103001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.103169 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.103699 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.103708 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.103990 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.104185 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.104499 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":57,"spinlock_wait_cycles":53632,"thread_start_us":152,"threads_started":1}
I20260619 11:05:27.106791 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.61@KRBTEST.COM for oryx/127.16.61.61@KRBTEST.COM
I20260619 11:05:27.114374 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.107264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.0.0.1:49184)
0619 11:05:27.107354 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.107368 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.107443 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.107827 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.107830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.107979 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.108051 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.108056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.108725 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.108731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.109220 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.109226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.110081 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.110084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.110196 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.110548 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.110558 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.111623 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.112999 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.113005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.113007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.113182 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.113467 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.113470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.113472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.113547 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.113878 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.113880 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.113954 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.114064 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.114183 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":4}
I20260619 11:05:27.114378 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.107321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.16.61.61:44913)
0619 11:05:27.107620 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.107623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.107632 (+ 9us) server_negotiation.cc:407] Connection header received
0619 11:05:27.107655 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.107656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.107729 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.107819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.108153 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.108620 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.109325 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.109975 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.110084 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.111778 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.111781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.111784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.111804 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.112898 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.113272 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.113278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.113280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.113373 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.113636 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.113639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.113641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.113785 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.113882 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.114104 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.114231 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":252}
I20260619 11:05:27.116832 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Bootstrap starting.
I20260619 11:05:27.118786 23286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:27.119405 23286 log.cc:824] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:27.121131 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: No bootstrap required, opened a new log
I20260619 11:05:27.123963 23286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } }
I20260619 11:05:27.124145 23286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:27.124176 23286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19087d9aa3d849c1bad2e98b83a56c72, State: Initialized, Role: FOLLOWER
I20260619 11:05:27.124565 23286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } }
I20260619 11:05:27.124935 23292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } } }
I20260619 11:05:27.125046 23292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:05:27.125562 23286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:05:27.129627 23303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:05:27.129719 23303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:05:27.131423 23218 master_runner.cc:428] Detected that this master 19087d9aa3d849c1bad2e98b83a56c72 is joining an existing cluster
I20260619 11:05:27.131484 23218 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.61.61:44913
I20260619 11:05:27.141202 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.133644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.16.61.62:45373)
0619 11:05:27.133759 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.133763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.133776 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:27.133813 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.133816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.133858 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.133942 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.134445 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.135235 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.136168 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.136397 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.136513 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.137647 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.137650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.137653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.137676 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.139393 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.140004 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.140010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.140014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.140102 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.140381 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.140386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.140388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.140563 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.140669 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.140928 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.141041 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:05:27.141347 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.133289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:55492)
0619 11:05:27.133611 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.133627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.133713 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.133983 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.133985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.134143 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.134281 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.134287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.135349 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.135352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.136051 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.136058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.136526 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.136530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.136636 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.137117 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.137135 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.137517 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.139508 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.139512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.139514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.139891 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.140202 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.140205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.140207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.140287 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.140659 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.140662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.140749 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.140869 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.141006 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260619 11:05:27.141598 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.133672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.16.61.61:44913)
0619 11:05:27.133771 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.133776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.134247 (+ 471us) server_negotiation.cc:407] Connection header received
0619 11:05:27.134333 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.134338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.134384 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.134451 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.134873 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.135538 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.136788 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.137058 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.137157 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.138300 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.138303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.138306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.138331 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.140053 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.140507 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.140510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.140512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.140582 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.140845 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.140850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.140853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.141034 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.141120 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.141397 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.141510 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":35}
I20260619 11:05:27.141919 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.133364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.0.0.1:49194)
0619 11:05:27.134148 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.134161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.134262 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.134545 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.134547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.134690 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.134766 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.134770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.135698 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.135701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.136670 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.136681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.137161 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.137165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.137302 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.137767 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.137781 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.138159 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.140156 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.140159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.140161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.140402 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.140676 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.140679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.140681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.140753 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.141151 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.141153 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.141219 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.141711 (+ 492us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.141813 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":51,"thread_start_us":112,"threads_started":1}
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.62@KRBTEST.COM for oryx/127.16.61.61@KRBTEST.COM
I20260619 11:05:27.153887 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.145568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.0.0.1:49210)
0619 11:05:27.145854 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.145901 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.146031 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.146314 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.146318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.146618 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.146705 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.146711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.147624 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.147626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.148263 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.148272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.148799 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.148802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.148873 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.149338 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.149349 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.150764 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.152527 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.152535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.152539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.152746 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.153007 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.153009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.153010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.153080 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.153443 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.153448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.153536 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.153643 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.153771 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260619 11:05:27.153946 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.145597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.16.61.61:44913)
0619 11:05:27.145694 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.145698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.145978 (+ 280us) server_negotiation.cc:407] Connection header received
0619 11:05:27.146153 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.146156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.146201 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.146281 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.146808 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.147526 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.148373 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.148692 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.148782 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.150891 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.150895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.150898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.150920 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.152427 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.152834 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.152837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.152839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.152919 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.153161 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.153164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.153166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.153350 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.153441 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.153707 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.153804 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:05:27.155330 22754 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.61.61:44913: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72"
member_type: NON_VOTER
last_known_addr {
host: "127.16.61.61"
port: 44913
}
attrs {
promote: true
}
}
dest_uuid: "9ca84261526b4b5da2e43f353fd49c8f"
cas_config_opid_index: -1
I20260619 11:05:27.155961 22754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } }
I20260619 11:05:27.157402 23313 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 19087d9aa3d849c1bad2e98b83a56c72 (127.16.61.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } }
I20260619 11:05:27.158200 23260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:27.159101 23313 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.61.61:44913
I20260619 11:05:27.159341 23313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } }
I20260619 11:05:27.159438 23313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:27.159653 23260 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Refusing update from remote peer 9ca84261526b4b5da2e43f353fd49c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 11:05:27.160244 23218 master.cc:537] Master@127.16.61.61:44913 shutting down...
I20260619 11:05:27.160449 23312 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:05:27.162220 22740 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 11:05:27.163439 22740 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f -> Peer 19087d9aa3d849c1bad2e98b83a56c72 (127.16.61.61:44913): Couldn't send request to peer 19087d9aa3d849c1bad2e98b83a56c72. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 11:05:27.164919 23218 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:27.165164 23218 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:27.165247 23218 tablet_replica.cc:330] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: stopping tablet replica
I20260619 11:05:27.170091 23218 master.cc:552] Master@127.16.61.61:44913 shutdown complete.
I20260619 11:05:27.170140 23218 master_runner.cc:305] Clearing existing system tablet
I20260619 11:05:27.170368 23218 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:27.170423 23218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
WARNING: no policy specified for oryx/127.16.61.60@KRBTEST.COM; defaulting to no policy
I20260619 11:05:27.171749 23218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.60@KRBTEST.COM" created.
I20260619 11:05:27.172933 23318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.173139 23218 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:27.173247 23218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:27.173278 23218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
uuid: "19087d9aa3d849c1bad2e98b83a56c72"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "2777c9fb579dd44cd45c5bc6d49d53fb"
server_key_iv: "c3d7ee91129475cabdd6564bf349d1ed"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.175189 23218 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:27.177415 23218 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 11:05:27.177498 23218 log.cc:1196] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 11:05:27.177726 23218 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Deleting consensus metadata
I20260619 11:05:27.177882 23218 master_runner.cc:315] Copying system tablet from 127.16.61.62:45373
I20260619 11:05:27.178900 23218 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: tablet copy: Beginning tablet copy session from remote peer at address 127.16.61.62:45373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.60.keytab.
Entry for principal oryx/127.16.61.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.60.keytab.
I20260619 11:05:27.190271 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.60:37719
--webserver_interface=127.16.61.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.60.keytab
--principal=oryx/127.16.61.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:45373,127.16.61.61:44913,127.16.61.60:37719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:05:27.192235 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.181660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.16.61.62:45373)
0619 11:05:27.181843 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.181848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.182149 (+ 301us) server_negotiation.cc:407] Connection header received
0619 11:05:27.182354 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.182374 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.182438 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.182539 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.183725 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.184787 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.186568 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.186914 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.186996 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.188446 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.188450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.188453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.188476 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.190367 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.190896 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.190902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.190907 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.190993 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.191304 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.191310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.191314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.191547 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.191622 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.191980 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.192086 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 11:05:27.192342 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.181613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:55498)
0619 11:05:27.182035 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.182050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.182219 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.183071 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.183076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.183400 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.183540 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.183548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.185016 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.185019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.186410 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.186424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.187147 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.187150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.187230 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.187810 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.187821 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.188238 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.190496 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.190502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.190507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.190768 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.191106 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.191112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.191117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.191207 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.191712 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.191715 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.191807 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.191925 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.192058 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":79,"threads_started":1}
I20260619 11:05:27.192756 22774 tablet_copy_service.cc:140] P 9ca84261526b4b5da2e43f353fd49c8f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 19087d9aa3d849c1bad2e98b83a56c72 ({username='oryx', principal='oryx/127.16.61.61@KRBTEST.COM'} at 127.0.0.1:55498)
I20260619 11:05:27.192931 22774 tablet_copy_service.cc:161] P 9ca84261526b4b5da2e43f353fd49c8f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 19087d9aa3d849c1bad2e98b83a56c72 at {username='oryx', principal='oryx/127.16.61.61@KRBTEST.COM'} at 127.0.0.1:55498: session id = 19087d9aa3d849c1bad2e98b83a56c72-00000000000000000000000000000000
I20260619 11:05:27.194583 22774 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 11:05:27.196192 23218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 11:05:27.199030 23218 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: tablet copy: Starting download of 0 data blocks...
I20260619 11:05:27.199226 23218 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: tablet copy: Starting download of 1 WAL segments...
I20260619 11:05:27.200973 23218 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 11:05:27.202943 22774 tablet_copy_service.cc:342] P 9ca84261526b4b5da2e43f353fd49c8f: Request end of tablet copy session 19087d9aa3d849c1bad2e98b83a56c72-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.61.61@KRBTEST.COM'} at 127.0.0.1:55498
I20260619 11:05:27.203061 22774 tablet_copy_service.cc:433] P 9ca84261526b4b5da2e43f353fd49c8f: ending tablet copy session 19087d9aa3d849c1bad2e98b83a56c72-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 19087d9aa3d849c1bad2e98b83a56c72
I20260619 11:05:27.210698 23218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:27.212252 23327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.212780 23330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.216121 23328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:27.216557 23218 server_base.cc:1034] running on GCE node
I20260619 11:05:27.216717 23218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:27.217000 23218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:27.218151 23218 hybrid_clock.cc:648] HybridClock initialized: now 1781867127218129 us; error 37 us; skew 500 ppm
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:27.219710 23218 init.cc:377] Logged in from keytab as oryx/127.16.61.61@KRBTEST.COM (short username oryx)
I20260619 11:05:27.220137 23218 webserver.cc:460] Webserver started at http://127.16.61.61:35409/ using document root <none> and password file <none>
I20260619 11:05:27.220314 23218 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:27.220356 23218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:27.221613 23218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:27.222558 23337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.222826 23218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:27.222900 23218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
uuid: "19087d9aa3d849c1bad2e98b83a56c72"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "2777c9fb579dd44cd45c5bc6d49d53fb"
server_key_iv: "c3d7ee91129475cabdd6564bf349d1ed"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.222987 23218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:27.241588 23218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:27.243523 23218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:27.248473 23218 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.61:44913
I20260619 11:05:27.248570 23389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.61:44913 every 8 connection(s)
I20260619 11:05:27.249027 23218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
I20260619 11:05:27.250975 23390 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:05:27.251994 23390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Bootstrap starting.
I20260619 11:05:27.261715 23390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:05:27.262017 23390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Bootstrap complete.
I20260619 11:05:27.262568 23390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } }
I20260619 11:05:27.262705 23390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19087d9aa3d849c1bad2e98b83a56c72, State: Initialized, Role: LEARNER
I20260619 11:05:27.262802 23390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } }
I20260619 11:05:27.263123 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } }
I20260619 11:05:27.263314 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:05:27.263223 23390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:27.266681 23404 catalog_manager.cc:1259] Loaded cluster ID: ed7a528273cd4b0e842c3ac9f472b358
I20260619 11:05:27.266783 23404 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: loading cluster ID for follower catalog manager: success
I20260619 11:05:27.268893 23404 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: acquiring CA information for follower catalog manager: success
I20260619 11:05:27.270246 23404 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 11:05:27.316478 23326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:27.316740 23326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:27.316818 23326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:27.320720 23326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:27.320796 23326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:27.320847 23326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:27.320899 23326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:27.320936 23326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:27.326227 23326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:45373,127.16.61.61:44913,127.16.61.60:37719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.60.keytab
--principal=oryx/127.16.61.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.60:37719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.61.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:27.327176 23326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:27.328153 23326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:27.335995 23410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.336079 23409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.335986 23412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:27.336210 23326 server_base.cc:1034] running on GCE node
I20260619 11:05:27.336809 23326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:27.337838 23326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:27.339046 23326 hybrid_clock.cc:648] HybridClock initialized: now 1781867127339038 us; error 31 us; skew 500 ppm
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:27.341531 23326 init.cc:377] Logged in from keytab as oryx/127.16.61.60@KRBTEST.COM (short username oryx)
I20260619 11:05:27.342291 23326 webserver.cc:460] Webserver started at http://127.16.61.60:45865/ using document root <none> and password file <none>
I20260619 11:05:27.342792 23326 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:27.342877 23326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:27.343091 23326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:27.344990 23326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data/instance:
uuid: "6bc8c885cc324a96aea7d4153b21a2ff"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "66dd70e470e669c6b65f7929d256f05e"
server_key_iv: "6f7c475d6204aa2846cb6dc73ee29982"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.345515 23326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal/instance:
uuid: "6bc8c885cc324a96aea7d4153b21a2ff"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "66dd70e470e669c6b65f7929d256f05e"
server_key_iv: "6f7c475d6204aa2846cb6dc73ee29982"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.348976 23326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:05:27.351044 23419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.352185 23326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:27.352310 23326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
uuid: "6bc8c885cc324a96aea7d4153b21a2ff"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "66dd70e470e669c6b65f7929d256f05e"
server_key_iv: "6f7c475d6204aa2846cb6dc73ee29982"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.352412 23326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:27.374629 23326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:27.377539 23326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:27.377719 23326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:27.384277 23326 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.60:37719
I20260619 11:05:27.384552 23471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.60:37719 every 8 connection(s)
I20260619 11:05:27.385221 23326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
I20260619 11:05:27.387830 23472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:27.388715 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 23326
I20260619 11:05:27.392637 23472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.61.60@KRBTEST.COM
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.60@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:27.403115 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.389953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.16.61.60:37719)
0619 11:05:27.390300 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.390313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.390345 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:05:27.390989 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.391005 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.391301 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.391626 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.392436 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.393446 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.394424 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.394958 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.395074 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.398188 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.398195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.398207 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.398235 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.400260 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.400682 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.400688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.400693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.400848 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.401325 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.401327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.401329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.401682 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.401864 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.402157 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.402402 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260619 11:05:27.404260 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.394037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:55512)
0619 11:05:27.394425 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.394461 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.394588 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.394870 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.394883 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.395327 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.395489 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.395495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.396618 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.396623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.397629 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.397642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.398292 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.398295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.398434 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.399169 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.399192 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.400698 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.402539 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.402549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.402554 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.402799 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.403083 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.403086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.403088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.403196 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.403628 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.403632 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.403788 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.403923 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.404081 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":129,"threads_started":1}
I20260619 11:05:27.405445 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.394239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.16.61.62:45373)
0619 11:05:27.394360 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.394365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.394534 (+ 169us) server_negotiation.cc:407] Connection header received
0619 11:05:27.394691 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.394694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.394746 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.394827 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.395633 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.396433 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.397772 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.398182 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.398291 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.400828 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.400834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.400836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.400863 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.402442 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.402889 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.402893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.402894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.402995 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.403310 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.403313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.403315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.403511 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.403565 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.403975 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.404081 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:05:27.406219 23472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.60@KRBTEST.COM for oryx/127.16.61.61@KRBTEST.COM
I20260619 11:05:27.413484 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.406609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.0.0.1:49212)
0619 11:05:27.406934 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.406946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.407001 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.407192 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.407194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.407321 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.407398 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.407402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.408261 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.408263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.408782 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.408790 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.409165 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.409167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.409232 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.409661 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.409672 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.410723 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.412386 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.412389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.412391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.412562 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.412787 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.412789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.412790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.412854 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.413155 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.413157 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.413220 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.413296 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.413383 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272}
I20260619 11:05:27.413625 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.406738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.16.61.61:44913)
0619 11:05:27.406969 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.406973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.406985 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:27.407061 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.407063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.407098 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.407168 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.407491 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.408171 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.408880 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.409080 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.409151 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.410866 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.410869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.410871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.410891 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.412280 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.412641 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.412643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.412645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.412715 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.412935 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.412936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.412937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.413084 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.413155 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.413367 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.413455 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260619 11:05:27.414636 23472 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.60@KRBTEST.COM for oryx/127.16.61.60@KRBTEST.COM
I20260619 11:05:27.421684 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.415075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.0.0.1:45248)
0619 11:05:27.415176 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.415192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.415261 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.415450 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.415452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.415569 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.415631 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.415635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.416700 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.416703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.417182 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.417189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.417571 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.417573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.417626 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.417971 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.417981 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.419013 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.420602 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.420604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.420605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.420776 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.420997 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.420999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.421000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.421064 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.421338 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.421340 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.421407 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.421484 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.421565 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 11:05:27.421715 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.415135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.16.61.60:37719)
0619 11:05:27.415232 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.415235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.415245 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:05:27.415328 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.415330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.415360 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.415426 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.415789 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.416249 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.417272 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.417486 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.417555 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.419125 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.419127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.419129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.419148 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.420513 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.420854 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.420856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.420857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.420927 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.421134 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.421136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.421137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.421268 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.421334 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.421532 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.421608 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:05:27.424386 23472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Bootstrap starting.
I20260619 11:05:27.426549 23472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:27.427217 23472 log.cc:824] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:27.428951 23472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: No bootstrap required, opened a new log
I20260619 11:05:27.432626 23472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } }
I20260619 11:05:27.432849 23472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:27.432909 23472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc8c885cc324a96aea7d4153b21a2ff, State: Initialized, Role: FOLLOWER
I20260619 11:05:27.433344 23472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } }
I20260619 11:05:27.433776 23478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } } }
I20260619 11:05:27.433930 23478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:05:27.434504 23472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:05:27.438778 23489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:05:27.438853 23489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:05:27.441954 23326 master_runner.cc:428] Detected that this master 6bc8c885cc324a96aea7d4153b21a2ff is joining an existing cluster
I20260619 11:05:27.442001 23326 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.61.60:37719
I20260619 11:05:27.452677 22945 heartbeater.cc:503] Master 127.16.61.62:45373 was elected leader, sending a full tablet report...
I20260619 11:05:27.455955 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 22816
I20260619 11:05:27.462182 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:42999
--local_ip_for_outbound_sockets=127.16.61.1
--tserver_master_addrs=127.16.61.62:45373
--webserver_port=34871
--webserver_interface=127.16.61.1
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab
--principal=oryx/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913,127.16.61.60:37719 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:05:27.472151 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.443830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.0.0.1:49220)
0619 11:05:27.444586 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.444601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.444671 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.444986 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.444989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.445213 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.445333 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.445343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.446366 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.446369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.447328 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.447336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.448030 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.448034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.448117 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.448758 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.448783 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.468248 (+ 19465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.470395 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.470402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.470405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.470658 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.470978 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.470985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.470989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.471074 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.471560 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.471565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.471689 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.471801 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.471960 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":64,"spinlock_wait_cycles":7040,"thread_start_us":104,"threads_started":1}
I20260619 11:05:27.472733 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.444329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.16.61.61:44913)
0619 11:05:27.444812 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.444816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.444827 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:05:27.444853 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.444855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.444886 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.444970 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.445455 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.446244 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.447448 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.447801 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.447883 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.468443 (+ 20560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.468447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.468449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.468476 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.470224 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.470773 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.470777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.470780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.470869 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.471181 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.471183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.471185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.471431 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.471517 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.472479 (+ 962us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.472592 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407}
I20260619 11:05:27.472749 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.443792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:55514)
0619 11:05:27.444072 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.444087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.444171 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.444688 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.444692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.444915 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.445059 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.445067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.446130 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.446133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.447929 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.447939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.448517 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.448521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.448626 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.468225 (+ 19599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.468241 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.468649 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.471191 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.471194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.471195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.471399 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.471724 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.471728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.471730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.471812 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.472257 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.472260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.472352 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.472447 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.472578 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":8,"thread_start_us":110,"threads_started":1}
I20260619 11:05:27.473264 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.444208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.16.61.62:45373)
0619 11:05:27.444461 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.444466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.444482 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:05:27.444520 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.444523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.444566 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.444656 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.445194 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.446003 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.448064 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.448409 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.448510 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.469119 (+ 20609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.469124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.469127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.469154 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.471085 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.471512 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.471515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.471517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.471595 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.471930 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.471934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.471935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.472124 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.472215 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.473018 (+ 803us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.473129 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 11:05:27.477391 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.444243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.16.61.60:37719)
0619 11:05:27.444335 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.444339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.445231 (+ 892us) server_negotiation.cc:407] Connection header received
0619 11:05:27.445388 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.445393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.445434 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.445479 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.445954 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.446638 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.447752 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.448066 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.448126 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.473881 (+ 25755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.473884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.473886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.473905 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.475558 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.476202 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.476205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.476206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.476280 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.476597 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.476600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.476602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.476774 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.476840 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.477138 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.477237 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":31,"spinlock_wait_cycles":2944}
I20260619 11:05:27.477847 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.444338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.0.0.1:45250)
0619 11:05:27.445161 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.445173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.445283 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.445578 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.445580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.445756 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.445850 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.445857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.446744 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.446747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.447529 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.447536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.448334 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.448340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.448735 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.449149 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.449162 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.473763 (+ 24601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.475768 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.475771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.475773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.476093 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.476408 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.476412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.476415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.476506 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.476898 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.476901 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.476979 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.477555 (+ 576us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.477678 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":84,"threads_started":1}
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.62@KRBTEST.COM for oryx/127.16.61.60@KRBTEST.COM
I20260619 11:05:27.485021 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.477734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.16.61.60:37719)
0619 11:05:27.477817 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.477832 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.477849 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:05:27.477912 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.477915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.477974 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.478046 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.478550 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.479137 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.479963 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.480190 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.480295 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.482104 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.482107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.482110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.482138 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.483449 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.483944 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.483950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.483952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.484024 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.484301 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.484304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.484306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.484455 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.484555 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.484798 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.484887 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:05:27.485028 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.477535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.0.0.1:45254)
0619 11:05:27.477707 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.477723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.477799 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.478160 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.478166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.478335 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.478428 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.478435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.479246 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.479253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.479853 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.479861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.480296 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.480299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.480402 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.480833 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.480844 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.481979 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.483562 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.483567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.483569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.483850 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.484122 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.484125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.484127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.484208 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.484549 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.484552 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.484644 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.484751 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.484888 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 11:05:27.486105 22754 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.61.60:37719: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff"
member_type: NON_VOTER
last_known_addr {
host: "127.16.61.60"
port: 37719
}
attrs {
promote: true
}
}
dest_uuid: "9ca84261526b4b5da2e43f353fd49c8f"
cas_config_opid_index: 5
I20260619 11:05:27.486650 22754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } }
I20260619 11:05:27.487761 23312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.488304 23446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:27.489174 23312 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.61.60:37719
I20260619 11:05:27.489364 23446 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 FOLLOWER]: Refusing update from remote peer 9ca84261526b4b5da2e43f353fd49c8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 11:05:27.489323 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.489420 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:27.489719 23326 master.cc:537] Master@127.16.61.60:37719 shutting down...
I20260619 11:05:27.490026 23313 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:05:27.491699 22738 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f -> Peer 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60:37719): Couldn't send request to peer 6bc8c885cc324a96aea7d4153b21a2ff. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 11:05:27.494293 23326 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:27.494552 23326 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:27.494652 23326 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: stopping tablet replica
I20260619 11:05:27.496013 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.487736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.0.0.1:49236)
0619 11:05:27.487886 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.487900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.487964 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.491209 (+ 3245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.491212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.491230 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:27.491317 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.491339 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.492476 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.492480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.493708 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.493720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.495614 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.495617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.495725 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.495772 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.495845 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.495906 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 11:05:27.496487 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.487810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.16.61.61:44913)
0619 11:05:27.490403 (+ 2593us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.490407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.490421 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:05:27.490462 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.490465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.490517 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.490604 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:27.491469 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.492262 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.493852 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.494511 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.494594 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.494741 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.496285 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.496367 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2497}
I20260619 11:05:27.497205 23364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 LEARNER]: Refusing update from remote peer 9ca84261526b4b5da2e43f353fd49c8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 11:05:27.497623 23313 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 11:05:27.499431 23364 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.500588 23326 master.cc:552] Master@127.16.61.60:37719 shutdown complete.
I20260619 11:05:27.500623 23326 master_runner.cc:305] Clearing existing system tablet
I20260619 11:05:27.500842 23326 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:27.500883 23326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:27.501968 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca84261526b4b5da2e43f353fd49c8f. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.502091 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:05:27.502203 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: NON_VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: true } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.502279 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: This master's current role is: LEARNER
I20260619 11:05:27.502502 23326 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:27.502764 23313 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: attempting to promote NON_VOTER 19087d9aa3d849c1bad2e98b83a56c72 to VOTER
I20260619 11:05:27.503211 23313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } }
I20260619 11:05:27.504531 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.504216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.0.0.1:45264)
0619 11:05:27.504396 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.504464 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260619 11:05:27.504766 23364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 LEARNER]: Refusing update from remote peer 9ca84261526b4b5da2e43f353fd49c8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 11:05:27.505225 23312 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 11:05:27.505656 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.505317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.0.0.1:45280)
0619 11:05:27.505548 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.505598 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260619 11:05:27.506198 22738 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f -> Peer 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60:37719): Couldn't send request to peer 6bc8c885cc324a96aea7d4153b21a2ff. Status: Network error: Client connection negotiation failed: client connection to 127.16.61.60:37719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 11:05:27.506801 23503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.506904 23312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19087d9aa3d849c1bad2e98b83a56c72 (127.16.61.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.507028 23326 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:27.507357 23326 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260619 11:05:27.507439 23326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
uuid: "6bc8c885cc324a96aea7d4153b21a2ff"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "66dd70e470e669c6b65f7929d256f05e"
server_key_iv: "6f7c475d6204aa2846cb6dc73ee29982"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.508038 23364 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19087d9aa3d849c1bad2e98b83a56c72 (127.16.61.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.509387 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.509486 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:05:27.509742 23326 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:27.510566 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.510669 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:27.510794 23312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.510872 23312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:27.512079 23326 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 11:05:27.512179 23326 log.cc:1196] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 11:05:27.512406 23326 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Deleting consensus metadata
I20260619 11:05:27.512581 23326 master_runner.cc:315] Copying system tablet from 127.16.61.62:45373
I20260619 11:05:27.513615 23326 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: tablet copy: Beginning tablet copy session from remote peer at address 127.16.61.62:45373
I20260619 11:05:27.521055 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.514355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.16.61.62:45373)
0619 11:05:27.514442 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.514446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.514458 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:27.514579 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.514582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.514622 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.514710 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.515126 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.515634 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.516579 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.516807 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.516912 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.517984 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.517988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.517990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.518012 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.519432 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.519929 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.519934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.519936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.520011 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.520277 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.520282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.520283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.520453 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.520556 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.520812 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.520911 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 11:05:27.521072 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.514062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:55522)
0619 11:05:27.514262 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.514277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.514367 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.514728 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.514730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.514933 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.515016 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.515022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.515886 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.515889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.516476 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.516487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.516916 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.516919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.517005 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.517531 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.517551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.517852 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.519560 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.519563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.519565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.519834 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.520101 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.520105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.520107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.520191 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.520557 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.520560 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.520653 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.520752 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.520898 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260619 11:05:27.521698 22774 tablet_copy_service.cc:140] P 9ca84261526b4b5da2e43f353fd49c8f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6bc8c885cc324a96aea7d4153b21a2ff ({username='oryx', principal='oryx/127.16.61.60@KRBTEST.COM'} at 127.0.0.1:55522)
I20260619 11:05:27.521791 22774 tablet_copy_service.cc:161] P 9ca84261526b4b5da2e43f353fd49c8f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6bc8c885cc324a96aea7d4153b21a2ff at {username='oryx', principal='oryx/127.16.61.60@KRBTEST.COM'} at 127.0.0.1:55522: session id = 6bc8c885cc324a96aea7d4153b21a2ff-00000000000000000000000000000000
I20260619 11:05:27.522588 22774 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 11:05:27.523744 23326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 11:05:27.525383 23326 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: tablet copy: Starting download of 0 data blocks...
I20260619 11:05:27.525563 23326 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: tablet copy: Starting download of 1 WAL segments...
I20260619 11:05:27.526835 23326 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 11:05:27.528216 22774 tablet_copy_service.cc:342] P 9ca84261526b4b5da2e43f353fd49c8f: Request end of tablet copy session 6bc8c885cc324a96aea7d4153b21a2ff-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.61.60@KRBTEST.COM'} at 127.0.0.1:55522
I20260619 11:05:27.528306 22774 tablet_copy_service.cc:433] P 9ca84261526b4b5da2e43f353fd49c8f: ending tablet copy session 6bc8c885cc324a96aea7d4153b21a2ff-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6bc8c885cc324a96aea7d4153b21a2ff
I20260619 11:05:27.534083 23326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:27.535619 23514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:27.535983 23326 server_base.cc:1034] running on GCE node
W20260619 11:05:27.535920 23511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.540110 23512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:27.540319 23326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:27.540616 23326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:27.541752 23326 hybrid_clock.cc:648] HybridClock initialized: now 1781867127541738 us; error 34 us; skew 500 ppm
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:27.543241 23326 init.cc:377] Logged in from keytab as oryx/127.16.61.60@KRBTEST.COM (short username oryx)
I20260619 11:05:27.543712 23326 webserver.cc:460] Webserver started at http://127.16.61.60:41095/ using document root <none> and password file <none>
I20260619 11:05:27.543900 23326 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:27.543954 23326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:27.544955 23326 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:27.545855 23521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.546074 23326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:27.546145 23326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
uuid: "6bc8c885cc324a96aea7d4153b21a2ff"
format_stamp: "Formatted at 2026-06-19 11:05:27 on dist-test-slave-8vcz"
server_key: "66dd70e470e669c6b65f7929d256f05e"
server_key_iv: "6f7c475d6204aa2846cb6dc73ee29982"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.546226 23326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:27.556623 23326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:27.558553 23326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:27.563308 23326 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.60:37719
I20260619 11:05:27.563318 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.60:37719 every 8 connection(s)
I20260619 11:05:27.563930 23326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
I20260619 11:05:27.565284 23574 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:05:27.566098 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Bootstrap starting.
I20260619 11:05:27.574306 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:05:27.574589 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: Bootstrap complete.
I20260619 11:05:27.575039 23574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } }
I20260619 11:05:27.575171 23574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bc8c885cc324a96aea7d4153b21a2ff, State: Initialized, Role: LEARNER
I20260619 11:05:27.575292 23574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } }
I20260619 11:05:27.575603 23577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.575785 23577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: This master's current role is: LEARNER
I20260619 11:05:27.575795 23574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:27.578858 23588 catalog_manager.cc:1259] Loaded cluster ID: ed7a528273cd4b0e842c3ac9f472b358
I20260619 11:05:27.578948 23588 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: loading cluster ID for follower catalog manager: success
I20260619 11:05:27.580595 23588 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: acquiring CA information for follower catalog manager: success
I20260619 11:05:27.581718 23588 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 11:05:27.619431 23498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:27.619771 23498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:27.619831 23498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:27.625957 23498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:27.626044 23498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:27.626168 23498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:27.633776 23498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab
--principal=oryx/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:42999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=34871
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913,127.16.61.60:37719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:27.634864 23498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:27.636446 23498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:05:27.644351 23079 heartbeater.cc:503] Master 127.16.61.62:45373 was elected leader, sending a full tablet report...
W20260619 11:05:27.646368 23594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.646410 23593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.646407 23596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:27.646845 23498 server_base.cc:1034] running on GCE node
I20260619 11:05:27.647356 23498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:27.648044 23498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:27.649214 23498 hybrid_clock.cc:648] HybridClock initialized: now 1781867127649169 us; error 55 us; skew 500 ppm
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:27.651744 23498 init.cc:377] Logged in from keytab as oryx/127.16.61.1@KRBTEST.COM (short username oryx)
I20260619 11:05:27.652499 23498 webserver.cc:460] Webserver started at http://127.16.61.1:34871/ using document root <none> and password file <none>
I20260619 11:05:27.653023 23498 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:27.653123 23498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:27.656687 23498 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:27.658636 23603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.659912 23498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:27.660034 23498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "069869d675a740a09d0f034a0566ddd9"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "b3bc1fa38d2f918270dffb5c4fcfa1d4"
server_key_iv: "65817f9ec40ae8d98294290c5b7ae3ca"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.660534 23498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:27.690963 23498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:27.693632 23498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:27.693827 23498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:27.694379 23498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:27.695174 23498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:27.695245 23498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.695307 23498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:27.695361 23498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.704772 23498 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:42999
I20260619 11:05:27.704843 23716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:42999 every 8 connection(s)
I20260619 11:05:27.705709 23498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:27.713922 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 23498
I20260619 11:05:27.714102 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 22950
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.60@KRBTEST.COM
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.61@KRBTEST.COM
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:27.721752 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:36265
--local_ip_for_outbound_sockets=127.16.61.2
--tserver_master_addrs=127.16.61.62:45373
--webserver_port=40641
--webserver_interface=127.16.61.2
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab
--principal=oryx/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913,127.16.61.60:37719 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:05:27.722303 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.708018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:34487 (local address 127.16.61.61:44913)
0619 11:05:27.708139 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.708142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.709013 (+ 871us) server_negotiation.cc:407] Connection header received
0619 11:05:27.710143 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.710146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.710202 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.710252 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.711651 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.712470 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.713862 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.714201 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.714293 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.718000 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.718007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.718013 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.718044 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.719787 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.720414 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.720420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.720425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.720514 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.720943 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.720946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.720948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.721179 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.721253 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.721932 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.722059 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:05:27.723404 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.707419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:52703 (local address 127.16.61.62:45373)
0619 11:05:27.707551 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.707554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.709009 (+ 1455us) server_negotiation.cc:407] Connection header received
0619 11:05:27.710142 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.710146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.710198 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.710252 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.711653 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.712472 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.717103 (+ 4631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.717448 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.717535 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.719949 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.719953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.719957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.719987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.721680 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.722164 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.722167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.722169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.722248 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.722547 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.722550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.722553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.722740 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.722816 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.723162 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.723270 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:05:27.723680 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.708308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.16.61.1:34487)
0619 11:05:27.708858 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.708914 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.709917 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.710444 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.710449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.710905 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.711505 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.711515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.712623 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.712626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.713692 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.713706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.715583 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.715589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.715741 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.716254 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.716274 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.717835 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.719922 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.719931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.719948 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.720300 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.720629 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.720634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.720639 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.720814 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.721334 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.721341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.721638 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.722505 (+ 867us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.722664 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":291,"spinlock_wait_cycles":62336,"thread_start_us":135,"threads_started":1}
I20260619 11:05:27.724160 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.708308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.1:52703)
0619 11:05:27.708857 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.708914 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.709917 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.710444 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.710449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.710881 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.711507 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.711515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.712623 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.712626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.716964 (+ 4338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.716975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.717697 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.717701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.717814 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.718271 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.718293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.719778 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.721794 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.721799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.721804 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.722058 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.722355 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.722358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.722360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.722455 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.722885 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.722888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.722985 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.723548 (+ 563us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.723713 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":221,"spinlock_wait_cycles":23936,"thread_start_us":144,"threads_started":1}
I20260619 11:05:27.724587 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.709342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.16.61.1:33643)
0619 11:05:27.709611 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.709624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.709930 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.710441 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.710448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.710878 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.711495 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.711509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.712495 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.712499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.713335 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.713351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.713866 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.713871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.714079 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.715206 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.715220 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.717016 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.719987 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.719991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.719993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.720481 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.720800 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.720804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.720807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.720902 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.721405 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.721411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.721639 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.721826 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.722588 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":90,"spinlock_wait_cycles":20480,"thread_start_us":102,"threads_started":1}
I20260619 11:05:27.724946 23719 heartbeater.cc:347] Connected to a master server at 127.16.61.61:44913
I20260619 11:05:27.724939 23718 heartbeater.cc:347] Connected to a master server at 127.16.61.62:45373
I20260619 11:05:27.725195 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.709452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:33643 (local address 127.16.61.60:37719)
0619 11:05:27.709693 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.709707 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.709721 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:05:27.710124 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.710127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.710185 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.710266 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.711633 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.712357 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.713464 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.713706 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.713837 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.717177 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.717181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.717184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.717207 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.719886 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.720599 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.720603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.720605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.720688 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.721016 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.721021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.721025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.721253 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.721358 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.723969 (+ 2611us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.724105 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260619 11:05:27.725214 23719 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:27.725739 23719 heartbeater.cc:511] Master 127.16.61.61:44913 requested a full tablet report, sending...
I20260619 11:05:27.725855 23717 heartbeater.cc:347] Connected to a master server at 127.16.61.60:37719
I20260619 11:05:27.725919 23717 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:27.726063 23717 heartbeater.cc:511] Master 127.16.61.60:37719 requested a full tablet report, sending...
I20260619 11:05:27.725212 23718 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:27.726418 23718 heartbeater.cc:511] Master 127.16.61.62:45373 requested a full tablet report, sending...
I20260619 11:05:27.727147 22754 ts_manager.cc:194] Re-registered known tserver with Master: 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999)
I20260619 11:05:27.727214 23538 ts_manager.cc:194] Registered new tserver with Master: 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999)
I20260619 11:05:27.727458 23354 ts_manager.cc:194] Registered new tserver with Master: 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999)
I20260619 11:05:27.727933 22754 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:52703
W20260619 11:05:27.831732 23725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:27.831961 23725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:27.832000 23725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:27.835997 23725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:27.836046 23725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:27.836155 23725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:27.841079 23725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab
--principal=oryx/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:36265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=40641
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913,127.16.61.60:37719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:27.841964 23725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:27.843048 23725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:05:27.852118 23213 heartbeater.cc:503] Master 127.16.61.62:45373 was elected leader, sending a full tablet report...
W20260619 11:05:27.852134 23730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:27.852257 23725 server_base.cc:1034] running on GCE node
W20260619 11:05:27.852119 23733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:27.852516 23731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:27.852955 23725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:27.853605 23725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:27.854789 23725 hybrid_clock.cc:648] HybridClock initialized: now 1781867127854770 us; error 34 us; skew 500 ppm
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:27.857544 23725 init.cc:377] Logged in from keytab as oryx/127.16.61.2@KRBTEST.COM (short username oryx)
I20260619 11:05:27.858336 23725 webserver.cc:460] Webserver started at http://127.16.61.2:40641/ using document root <none> and password file <none>
I20260619 11:05:27.858866 23725 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:27.858939 23725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:27.862396 23725 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:27.864472 23740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.865707 23725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:27.865793 23725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "c6cd46533d1e402fb5661105f73741d6"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "4da5f03da00dc8782b3428995d0b2d83"
server_key_iv: "6686b4e371c119c4d2ccc276212dd197"
server_key_version: "encryptionkey@0"
I20260619 11:05:27.866250 23725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:27.900242 23725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:27.903285 23725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:27.903426 23725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:27.903965 23725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:27.904698 23725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:27.904742 23725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.904779 23725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:27.904810 23725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:27.913352 23725 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:36265
I20260619 11:05:27.913380 23853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:36265 every 8 connection(s)
I20260619 11:05:27.914233 23725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:27.918812 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 23725
I20260619 11:05:27.918982 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 23084
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.60@KRBTEST.COM
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.61@KRBTEST.COM
I20260619 11:05:27.927597 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:32829
--local_ip_for_outbound_sockets=127.16.61.3
--tserver_master_addrs=127.16.61.62:45373
--webserver_port=33027
--webserver_interface=127.16.61.3
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab
--principal=oryx/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913,127.16.61.60:37719 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:05:27 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:27.931878 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.917697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:53409 (local address 127.16.61.60:37719)
0619 11:05:27.917831 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.917834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.917846 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:27.918103 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.918106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.918163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.918236 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.919760 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.920640 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.922723 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.923185 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.923295 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.927326 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.927333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.927339 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.927374 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.929422 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930162 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.930169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.930172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.930260 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930651 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.930656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.930659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.930904 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.930979 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.931556 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.931707 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:27.931878 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.915949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:41315 (local address 127.16.61.61:44913)
0619 11:05:27.916062 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.916066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.917342 (+ 1276us) server_negotiation.cc:407] Connection header received
0619 11:05:27.918213 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.918218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.918265 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.918334 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.919881 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.920673 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.922713 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.923073 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.923137 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.927303 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.927306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.927309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.927347 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.929590 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930205 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.930210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.930212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.930300 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930667 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.930670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.930672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.930917 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.931001 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.931597 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.931722 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:05:27.932062 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.915848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:53007 (local address 127.16.61.62:45373)
0619 11:05:27.915961 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.915968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.916962 (+ 994us) server_negotiation.cc:407] Connection header received
0619 11:05:27.918126 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.918129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.918185 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.918236 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:27.919881 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.920681 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.922727 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.923052 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.923129 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.928433 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:27.928436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:27.928439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:27.928465 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:27.930330 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930790 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.930795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.930796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.930869 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:27.931171 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:27.931174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:27.931176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:27.931389 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:27.931495 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.931813 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.931921 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:05:27.932797 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.917406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.16.61.2:53409)
0619 11:05:27.917652 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.917665 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.917917 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.918382 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.918388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.918951 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.919564 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.919586 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.920824 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.920831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.922587 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.922600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.923491 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.923498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.923701 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.924449 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.924474 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.927110 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.929558 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.929563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.929577 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.930043 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.930372 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.930380 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.930543 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.931061 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.931065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.931316 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.932413 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.932519 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":240,"spinlock_wait_cycles":10880,"thread_start_us":62,"threads_started":1}
I20260619 11:05:27.932806 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.916323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.2:53007)
0619 11:05:27.916822 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.916859 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.917915 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.918385 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.918393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.918909 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.919741 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.919749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.920816 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.920819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.922577 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.922588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.923329 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.923333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.923624 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.924532 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.924548 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.928293 (+ 3745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.930431 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.930436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.930688 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.930977 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.930981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.931063 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.931505 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.931508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.931604 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.932188 (+ 584us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.932312 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":325,"spinlock_wait_cycles":2816,"thread_start_us":668,"threads_started":1}
I20260619 11:05:27.934010 23855 heartbeater.cc:347] Connected to a master server at 127.16.61.62:45373
I20260619 11:05:27.934083 23854 heartbeater.cc:347] Connected to a master server at 127.16.61.60:37719
I20260619 11:05:27.934237 23854 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:27.934237 23855 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:27.934759 23854 heartbeater.cc:511] Master 127.16.61.60:37719 requested a full tablet report, sending...
I20260619 11:05:27.934763 23855 heartbeater.cc:511] Master 127.16.61.62:45373 requested a full tablet report, sending...
I20260619 11:05:27.935824 22754 ts_manager.cc:194] Re-registered known tserver with Master: c6cd46533d1e402fb5661105f73741d6 (127.16.61.2:36265)
I20260619 11:05:27.935879 23538 ts_manager.cc:194] Registered new tserver with Master: c6cd46533d1e402fb5661105f73741d6 (127.16.61.2:36265)
I20260619 11:05:27.936545 22754 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:53007
I20260619 11:05:27.939738 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.916323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.16.61.2:41315)
0619 11:05:27.917230 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.917243 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.917915 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.918428 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.918431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.918909 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:27.919744 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.919749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.920816 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.920819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.922581 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.922588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.923329 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.923333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.923626 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.924643 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:27.924661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:27.927110 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:27.929703 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.929707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.929710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.930078 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.930417 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:27.930420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:27.930423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:27.930550 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:27.931069 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:27.931072 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:27.931313 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.931557 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.931900 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":524,"spinlock_wait_cycles":68352,"thread_start_us":163,"threads_started":1}
I20260619 11:05:27.940464 23856 heartbeater.cc:347] Connected to a master server at 127.16.61.61:44913
I20260619 11:05:27.940567 23856 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:27.940788 23856 heartbeater.cc:511] Master 127.16.61.61:44913 requested a full tablet report, sending...
I20260619 11:05:27.941587 23354 ts_manager.cc:194] Registered new tserver with Master: c6cd46533d1e402fb5661105f73741d6 (127.16.61.2:36265)
I20260619 11:05:27.992369 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.989028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.0.0.1:45292)
0619 11:05:27.989186 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:27.989209 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:27.989320 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:27.989642 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:27.989645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:27.989662 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:27.989759 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.989764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.990512 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.990517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.991244 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:27.991252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.991993 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.991997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:27.992077 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.992098 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:05:27.992162 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:05:27.992232 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 11:05:27.992370 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:27.989245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.16.61.60:37719)
0619 11:05:27.989413 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:27.989418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:27.989431 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:27.989486 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:27.989488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:27.989533 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:27.989629 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:27.989875 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.990400 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.991366 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:27.991881 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:27.991994 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:27.992136 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:27.992175 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 11:05:27.992230 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:05:27.993026 23548 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 11:05:27.993386 23577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca84261526b4b5da2e43f353fd49c8f. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:27.993561 23577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: This master's current role is: LEARNER
W20260619 11:05:28.040298 23861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:28.040516 23861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:28.040555 23861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:28.044467 23861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:28.044512 23861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:28.044600 23861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:28.050093 23861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab
--principal=oryx/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:32829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=33027
--tserver_master_addrs=127.16.61.62:45373,127.16.61.61:44913,127.16.61.60:37719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:28.051003 23861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:28.051997 23861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:28.061115 23866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:28.061137 23867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:28.061138 23869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:28.061273 23861 server_base.cc:1034] running on GCE node
I20260619 11:05:28.061712 23861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:28.062275 23861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:28.063481 23861 hybrid_clock.cc:648] HybridClock initialized: now 1781867128063442 us; error 50 us; skew 500 ppm
Jun 19 11:05:28 dist-test-slave-8vcz krb5kdc[22709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867128, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:28.069394 23861 init.cc:377] Logged in from keytab as oryx/127.16.61.3@KRBTEST.COM (short username oryx)
I20260619 11:05:28.070230 23861 webserver.cc:460] Webserver started at http://127.16.61.3:33027/ using document root <none> and password file <none>
I20260619 11:05:28.070767 23861 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:28.070879 23861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:28.074355 23861 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:28.076313 23876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:28.077572 23861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:28.077703 23861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "25b2f6b36d284fb7aab35e486f3b2015"
format_stamp: "Formatted at 2026-06-19 11:05:26 on dist-test-slave-8vcz"
server_key: "abe6ea2449dc5df098ce9f5dc231e86a"
server_key_iv: "8f57e1ca3d9124e0c908c5532f53675f"
server_key_version: "encryptionkey@0"
I20260619 11:05:28.078197 23861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:28.091269 23861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:28.094180 23861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:28.094367 23861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:28.094878 23861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:28.095651 23861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:28.095741 23861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:28.095808 23861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:28.095868 23861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:28.104423 23861 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:32829
I20260619 11:05:28.104489 23989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:32829 every 8 connection(s)
I20260619 11:05:28.105341 23861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:28.106896 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 23861
Jun 19 11:05:28 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867128, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.60@KRBTEST.COM
I20260619 11:05:28.118451 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.108793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.16.61.62:45373)
0619 11:05:28.109015 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.109018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.109362 (+ 344us) server_negotiation.cc:407] Connection header received
0619 11:05:28.109414 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.109420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.109472 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.109573 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.110300 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.110976 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.112155 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.112496 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.112597 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.114633 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.114636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.114639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.114663 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.116453 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.116912 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.116915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.116917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.116997 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.117276 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.117279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.117281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.117469 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.117589 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.118143 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.118263 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":102,"threads_started":1}
Jun 19 11:05:28 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867128, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
Jun 19 11:05:28 dist-test-slave-8vcz krb5kdc[22709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867128, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.61@KRBTEST.COM
I20260619 11:05:28.121569 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.107012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:51057 (local address 127.16.61.60:37719)
0619 11:05:28.107140 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.107144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.108174 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 11:05:28.109388 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.109391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.109447 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.109502 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.110697 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.111415 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.113146 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.113500 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.113596 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.116445 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.116450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.116454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.116492 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.118196 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.118906 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.118909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.118911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.118994 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.119398 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.119402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.119403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.119843 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.119935 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.121285 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.121401 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:05:28.123211 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.107474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.16.61.3:51057)
0619 11:05:28.108018 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:28.108072 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:28.109146 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:28.109805 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:28.109811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:28.110139 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:28.110566 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.110579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.111555 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.111561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.113004 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.113016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.113739 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.113743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.113954 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.114524 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:28.114543 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:28.116279 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:28.118437 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:28.118442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:28.118456 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:28.118805 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:28.119103 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:28.119106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:28.119107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:28.119281 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:28.120739 (+ 1458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:28.120745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:28.121035 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 11:05:28.121775 (+ 740us) client_negotiation.cc:240] Negotiation successful
0619 11:05:28.122116 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":163,"threads_started":1}
I20260619 11:05:28.124456 23990 heartbeater.cc:347] Connected to a master server at 127.16.61.60:37719
I20260619 11:05:28.121209 22754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:05:28.124706 23990 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:28.125277 23990 heartbeater.cc:511] Master 127.16.61.60:37719 requested a full tablet report, sending...
W20260619 11:05:28.125777 22754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:28.126454 23538 ts_manager.cc:194] Registered new tserver with Master: 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829)
I20260619 11:05:28.127327 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.107734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:52201 (local address 127.16.61.61:44913)
0619 11:05:28.107852 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.107856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.111615 (+ 3759us) server_negotiation.cc:407] Connection header received
0619 11:05:28.111744 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.111747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.111791 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.111862 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.113105 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.113967 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.117197 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.117532 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.117601 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.122016 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.122019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.122021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.122043 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.124269 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.125458 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.125462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.125463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.125543 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.126377 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.126380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.126382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.126583 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.126662 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.127061 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.127166 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:05:28.128057 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.107474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.16.61.3:52201)
0619 11:05:28.111521 (+ 4047us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:28.111534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:28.111626 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:28.112003 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:28.112007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:28.112846 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:28.112965 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.112975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.114116 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.114119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.115030 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.115042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.118253 (+ 3211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.118256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.118375 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.120305 (+ 1930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:28.120328 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:28.121875 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:28.124612 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:28.124615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:28.124617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:28.125029 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:28.125648 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:28.125651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:28.125653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:28.125733 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:28.126784 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:28.126789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:28.126884 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 11:05:28.127688 (+ 804us) client_negotiation.cc:240] Negotiation successful
0619 11:05:28.127828 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3935,"mutex_wait_us":369,"thread_start_us":95,"threads_started":1}
I20260619 11:05:28.128324 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.110978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.16.61.61:44913)
0619 11:05:28.111205 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.111210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.113560 (+ 2350us) server_negotiation.cc:407] Connection header received
0619 11:05:28.113601 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.113603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.113647 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.113726 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.116622 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.117469 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.120959 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.121322 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.121412 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.124092 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.124096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.124098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.124122 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.125999 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.126710 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.126713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.126715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.126803 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.127114 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.127118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.127119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.127338 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.127458 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.128000 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.128122 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260619 11:05:28.128798 23992 heartbeater.cc:347] Connected to a master server at 127.16.61.61:44913
I20260619 11:05:28.128880 23992 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:28.129037 23992 heartbeater.cc:511] Master 127.16.61.61:44913 requested a full tablet report, sending...
I20260619 11:05:28.129714 23354 ts_manager.cc:194] Registered new tserver with Master: 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829)
I20260619 11:05:28.130784 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.109332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.16.61.60:37719)
0619 11:05:28.112175 (+ 2843us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.112179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.112190 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:05:28.112221 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.112225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.112266 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.112345 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.114415 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.115175 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.118640 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.119011 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.119097 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.121522 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.121525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.121528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.121547 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.126060 (+ 4513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.127505 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.127510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.127514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.127596 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.128615 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.128619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.128621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.128868 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.129017 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.129354 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.130638 (+ 1284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2778,"thread_start_us":70,"threads_started":1}
I20260619 11:05:28.130836 23502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: NON_VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: true } } }
I20260619 11:05:28.130966 23502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:28.131234 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.108619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:32991 (local address 127.16.61.62:45373)
0619 11:05:28.108742 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.108748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.111948 (+ 3200us) server_negotiation.cc:407] Connection header received
0619 11:05:28.112058 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.112061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.112110 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.112178 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.115171 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.116006 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.117723 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.118040 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.118112 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.127734 (+ 9622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.127737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.127740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.127762 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.129419 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.129935 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.129938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.129940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.130026 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.130327 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.130330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.130332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.130547 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.130645 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.130971 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.131088 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":1280}
I20260619 11:05:28.132040 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.108534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.3:32991)
0619 11:05:28.111871 (+ 3337us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:28.111886 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:28.111987 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:28.112263 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:28.112266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:28.112470 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:28.112608 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.112615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.116181 (+ 3566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.116185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.117077 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.117086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.118792 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.118795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.118874 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.119452 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:28.119467 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:28.127602 (+ 8135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:28.129534 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:28.129537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:28.129538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:28.129764 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:28.130139 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:28.130142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:28.130144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:28.130225 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:28.130690 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:28.130693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:28.130784 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 11:05:28.131386 (+ 602us) client_negotiation.cc:240] Negotiation successful
0619 11:05:28.131537 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3269,"thread_start_us":97,"threads_started":1}
I20260619 11:05:28.132570 23991 heartbeater.cc:347] Connected to a master server at 127.16.61.62:45373
I20260619 11:05:28.132650 23991 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:28.132795 23991 heartbeater.cc:511] Master 127.16.61.62:45373 requested a full tablet report, sending...
I20260619 11:05:28.133375 22752 ts_manager.cc:194] Re-registered known tserver with Master: 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829)
I20260619 11:05:28.134047 22752 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:32991
I20260619 11:05:28.135071 23502 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: attempting to promote NON_VOTER 6bc8c885cc324a96aea7d4153b21a2ff to VOTER
I20260619 11:05:28.135375 23502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } }
I20260619 11:05:28.136375 23364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Refusing update from remote peer 9ca84261526b4b5da2e43f353fd49c8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 11:05:28.136926 24008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 11:05:28.137369 23547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 LEARNER]: Refusing update from remote peer 9ca84261526b4b5da2e43f353fd49c8f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 11:05:28.137723 24008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 11:05:28.138731 24016 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:28.139159 23364 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:28.139921 23547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:28.140791 24016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:28.140894 24016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:28.141110 24016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:28.141212 24016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:28.141181 23502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:28.141317 23502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:28.141863 23577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:28.141975 23577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:05:28.142977 24018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ca84261526b4b5da2e43f353fd49c8f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:28.143096 24018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:05:28.150153 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.144507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:36265 (local address 127.0.0.1:41326)
0619 11:05:28.144716 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:28.144732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:28.145429 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:28.146058 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:28.146060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:28.146088 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:28.146250 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.146256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.147375 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.147380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.148649 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.148659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.149709 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.149713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.149809 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.149834 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:05:28.149924 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:05:28.150009 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":25,"spinlock_wait_cycles":3328,"thread_start_us":97,"threads_started":1}
I20260619 11:05:28.150209 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.145277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.16.61.2:36265)
0619 11:05:28.145593 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.145604 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.145626 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:05:28.145702 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.145708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.145953 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.146080 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:28.146375 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.147263 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.148773 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.149591 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.149694 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.149834 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.149952 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.150042 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
I20260619 11:05:28.150611 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.144310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:32829 (local address 127.0.0.1:55330)
0619 11:05:28.144460 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:28.144473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:28.144550 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:28.146056 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:28.146059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:28.146090 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:28.146268 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.146279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.148062 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.148066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.149216 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.149226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.150206 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.150210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.150305 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.150340 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 11:05:28.150391 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 11:05:28.150473 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":67,"spinlock_wait_cycles":7936}
I20260619 11:05:28.151149 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.144420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.16.61.3:32829)
0619 11:05:28.145640 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.145652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.145669 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:05:28.145719 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.145722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.145913 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.146072 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:28.146471 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.147941 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.149368 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.150077 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.150801 (+ 724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.150910 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.150948 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.151005 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143,"thread_start_us":101,"threads_started":1}
I20260619 11:05:28.151537 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.145003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.16.61.1:42999)
0619 11:05:28.145294 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.145306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.146393 (+ 1087us) server_negotiation.cc:407] Connection header received
0619 11:05:28.146602 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.146607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.146782 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.147342 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:28.147750 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.148809 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.149983 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.150783 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.150857 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.150974 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.151305 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.151396 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":119,"threads_started":1}
I20260619 11:05:28.151930 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.144816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42999 (local address 127.0.0.1:34432)
0619 11:05:28.146270 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:28.146283 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:28.146463 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:28.147503 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:28.147505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:28.147521 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:28.147629 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.147634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.148916 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.148922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.149850 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.149858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.151069 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.151072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.151155 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.151179 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:05:28.151741 (+ 562us) client_negotiation.cc:240] Negotiation successful
0619 11:05:28.151811 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1395,"mutex_wait_us":26,"thread_start_us":80,"threads_started":1}
I20260619 11:05:28.153411 23924 tablet_service.cc:1467] Processing CreateTablet for tablet e092d117c9264c73b6a99d47707a06d3 (DEFAULT_TABLE table=test-table [id=a55abdb04ec04fb89ad408b3ff7881c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:28.153326 23788 tablet_service.cc:1467] Processing CreateTablet for tablet e092d117c9264c73b6a99d47707a06d3 (DEFAULT_TABLE table=test-table [id=a55abdb04ec04fb89ad408b3ff7881c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:28.154175 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e092d117c9264c73b6a99d47707a06d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:28.154300 23788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e092d117c9264c73b6a99d47707a06d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:28.154379 23651 tablet_service.cc:1467] Processing CreateTablet for tablet e092d117c9264c73b6a99d47707a06d3 (DEFAULT_TABLE table=test-table [id=a55abdb04ec04fb89ad408b3ff7881c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:28.155213 23651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e092d117c9264c73b6a99d47707a06d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:28.161091 24024 tablet_bootstrap.cc:492] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: Bootstrap starting.
I20260619 11:05:28.162885 24025 tablet_bootstrap.cc:492] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: Bootstrap starting.
I20260619 11:05:28.163130 24024 tablet_bootstrap.cc:654] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:28.163991 24026 tablet_bootstrap.cc:492] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Bootstrap starting.
I20260619 11:05:28.164093 24024 log.cc:824] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:28.165138 24025 tablet_bootstrap.cc:654] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:28.165876 24025 log.cc:824] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:28.166030 24026 tablet_bootstrap.cc:654] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:28.166318 24024 tablet_bootstrap.cc:492] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: No bootstrap required, opened a new log
I20260619 11:05:28.166505 24024 ts_tablet_manager.cc:1397] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:05:28.166817 24026 log.cc:824] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:28.167936 24025 tablet_bootstrap.cc:492] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: No bootstrap required, opened a new log
I20260619 11:05:28.168128 24025 ts_tablet_manager.cc:1397] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:05:28.168615 24026 tablet_bootstrap.cc:492] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: No bootstrap required, opened a new log
I20260619 11:05:28.168776 24026 ts_tablet_manager.cc:1397] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:05:28.170536 24024 raft_consensus.cc:348] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.170804 24024 raft_consensus.cc:374] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:28.170864 24024 raft_consensus.cc:729] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b2f6b36d284fb7aab35e486f3b2015, State: Initialized, Role: FOLLOWER
I20260619 11:05:28.171304 24024 consensus_queue.cc:260] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.172277 24024 ts_tablet_manager.cc:1428] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:05:28.172967 24025 raft_consensus.cc:348] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.173214 24025 raft_consensus.cc:374] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:28.173272 24025 raft_consensus.cc:729] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cd46533d1e402fb5661105f73741d6, State: Initialized, Role: FOLLOWER
I20260619 11:05:28.173557 23991 heartbeater.cc:503] Master 127.16.61.62:45373 was elected leader, sending a full tablet report...
I20260619 11:05:28.173650 24025 consensus_queue.cc:260] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.174566 24025 ts_tablet_manager.cc:1428] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:05:28.174842 23855 heartbeater.cc:503] Master 127.16.61.62:45373 was elected leader, sending a full tablet report...
I20260619 11:05:28.177389 24026 raft_consensus.cc:348] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.177657 24026 raft_consensus.cc:374] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:28.177708 24026 raft_consensus.cc:729] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069869d675a740a09d0f034a0566ddd9, State: Initialized, Role: FOLLOWER
I20260619 11:05:28.178195 24026 consensus_queue.cc:260] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.179015 24026 ts_tablet_manager.cc:1428] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260619 11:05:28.179781 23718 heartbeater.cc:503] Master 127.16.61.62:45373 was elected leader, sending a full tablet report...
W20260619 11:05:28.208078 23720 tablet.cc:2369] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:28.219946 24031 raft_consensus.cc:482] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:28.220237 24031 raft_consensus.cc:504] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.221933 24031 leader_election.cc:288] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829), 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999)
I20260619 11:05:28.227401 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.222203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42999 (local address 127.16.61.2:45763)
0619 11:05:28.222729 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:28.222742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:28.222839 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:28.223105 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:28.223109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:28.223129 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:28.223253 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.223261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.224210 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.224215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.225987 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.226001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.226945 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.226948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.227059 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.227082 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:05:28.227151 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:05:28.227225 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":74}
I20260619 11:05:28.227655 24019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.222265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:51679 (local address 127.16.61.3:32829)
0619 11:05:28.222389 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.222394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.222503 (+ 109us) server_negotiation.cc:407] Connection header received
0619 11:05:28.222667 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.222670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.222743 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.222826 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:28.223217 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.224047 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.226280 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.226877 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.226967 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.227022 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.227458 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.227548 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 11:05:28.227798 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.222391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:45763 (local address 127.16.61.1:42999)
0619 11:05:28.222504 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.222509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.222798 (+ 289us) server_negotiation.cc:407] Connection header received
0619 11:05:28.222940 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.222943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.222999 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.223077 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:28.223363 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.224103 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.226132 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.226716 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.226817 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.226883 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.227568 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.227651 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:05:28.227409 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.222132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:32829 (local address 127.16.61.2:51679)
0619 11:05:28.222404 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:28.222440 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:28.222542 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:28.222946 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:28.222949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:28.222966 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:28.223091 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.223099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.224183 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.224187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.226005 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:28.226016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.227072 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.227075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:28.227169 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.227186 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:05:28.227258 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:05:28.227311 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":12}
I20260619 11:05:28.228461 23671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e092d117c9264c73b6a99d47707a06d3" candidate_uuid: "c6cd46533d1e402fb5661105f73741d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069869d675a740a09d0f034a0566ddd9" is_pre_election: true
I20260619 11:05:28.228509 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e092d117c9264c73b6a99d47707a06d3" candidate_uuid: "c6cd46533d1e402fb5661105f73741d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b2f6b36d284fb7aab35e486f3b2015" is_pre_election: true
I20260619 11:05:28.228785 23944 raft_consensus.cc:2454] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cd46533d1e402fb5661105f73741d6 in term 0.
I20260619 11:05:28.228811 23671 raft_consensus.cc:2454] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cd46533d1e402fb5661105f73741d6 in term 0.
I20260619 11:05:28.229277 23742 leader_election.cc:302] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b2f6b36d284fb7aab35e486f3b2015, c6cd46533d1e402fb5661105f73741d6; no voters:
I20260619 11:05:28.229743 24031 raft_consensus.cc:2781] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:28.229835 24031 raft_consensus.cc:482] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:28.229892 24031 raft_consensus.cc:3037] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:28.231258 24031 raft_consensus.cc:504] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.231657 24031 leader_election.cc:288] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [CANDIDATE]: Term 1 election: Requested vote from peers 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829), 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999)
I20260619 11:05:28.232117 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e092d117c9264c73b6a99d47707a06d3" candidate_uuid: "c6cd46533d1e402fb5661105f73741d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b2f6b36d284fb7aab35e486f3b2015"
I20260619 11:05:28.232250 23671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e092d117c9264c73b6a99d47707a06d3" candidate_uuid: "c6cd46533d1e402fb5661105f73741d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069869d675a740a09d0f034a0566ddd9"
I20260619 11:05:28.232241 23944 raft_consensus.cc:3037] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:28.232352 23671 raft_consensus.cc:3037] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:28.233533 23944 raft_consensus.cc:2454] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cd46533d1e402fb5661105f73741d6 in term 1.
I20260619 11:05:28.233533 23671 raft_consensus.cc:2454] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cd46533d1e402fb5661105f73741d6 in term 1.
I20260619 11:05:28.233814 23742 leader_election.cc:302] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b2f6b36d284fb7aab35e486f3b2015, c6cd46533d1e402fb5661105f73741d6; no voters:
I20260619 11:05:28.234007 24031 raft_consensus.cc:2781] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:28.234228 24031 raft_consensus.cc:686] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 1 LEADER]: Becoming Leader. State: Replica: c6cd46533d1e402fb5661105f73741d6, State: Running, Role: LEADER
I20260619 11:05:28.234586 24031 consensus_queue.cc:237] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } }
I20260619 11:05:28.238600 22754 catalog_manager.cc:5495] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6cd46533d1e402fb5661105f73741d6 (127.16.61.2). New cstate: current_term: 1 leader_uuid: "c6cd46533d1e402fb5661105f73741d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:28.270403 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.267328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.16.61.2:36265)
0619 11:05:28.267448 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.267455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.267472 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:05:28.267572 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.267575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.267626 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.267753 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:28.267917 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.268397 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.269222 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.269429 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.269509 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.269711 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:28.270075 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:28.270161 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.270241 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.270305 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:05:28.275942 23671 raft_consensus.cc:1261] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Refusing update from remote peer c6cd46533d1e402fb5661105f73741d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:28.275942 23944 raft_consensus.cc:1261] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 1 FOLLOWER]: Refusing update from remote peer c6cd46533d1e402fb5661105f73741d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:28.276705 24031 consensus_queue.cc:1035] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:28.276947 24033 consensus_queue.cc:1035] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:28.283772 24036 mvcc.cc:205] Tried to move back new op lower bound from 7298527757410689024 to 7298527757252046848. Current Snapshot: MvccSnapshot[applied={T|T < 7298527757410689024}]
I20260619 11:05:28.286275 24038 mvcc.cc:205] Tried to move back new op lower bound from 7298527757410689024 to 7298527757252046848. Current Snapshot: MvccSnapshot[applied={T|T < 7298527757410689024 or (T in {7298527757410689024})}]
I20260619 11:05:28.295590 22754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:28.295779 22754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:28.299396 22754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ca84261526b4b5da2e43f353fd49c8f: Sending DeleteTablet for 3 replicas of tablet e092d117c9264c73b6a99d47707a06d3
I20260619 11:05:28.300298 23924 tablet_service.cc:1514] Processing DeleteTablet for tablet e092d117c9264c73b6a99d47707a06d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:28 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:55330
I20260619 11:05:28.300503 23788 tablet_service.cc:1514] Processing DeleteTablet for tablet e092d117c9264c73b6a99d47707a06d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:28 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:41326
I20260619 11:05:28.300546 23651 tablet_service.cc:1514] Processing DeleteTablet for tablet e092d117c9264c73b6a99d47707a06d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:28 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:34432
I20260619 11:05:28.300910 24048 tablet_replica.cc:330] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: stopping tablet replica
I20260619 11:05:28.301247 24048 raft_consensus.cc:2229] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:28.301542 24048 raft_consensus.cc:2258] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:28.302762 24046 tablet_replica.cc:330] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: stopping tablet replica
I20260619 11:05:28.303015 24048 ts_tablet_manager.cc:1905] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:28.303012 24046 raft_consensus.cc:2229] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:28.303268 24046 raft_consensus.cc:2258] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:28.303953 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 22722
I20260619 11:05:28.306213 24048 ts_tablet_manager.cc:1918] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:28.306316 24048 log.cc:1196] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/e092d117c9264c73b6a99d47707a06d3
I20260619 11:05:28.306617 24048 ts_tablet_manager.cc:1939] T e092d117c9264c73b6a99d47707a06d3 P 069869d675a740a09d0f034a0566ddd9: Deleting consensus metadata
W20260619 11:05:28.312011 23605 connection.cc:535] server connection from 127.0.0.1:34432 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:05:28.312862 24046 ts_tablet_manager.cc:1905] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:28.314918 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.302470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.16.61.61:44913)
0619 11:05:28.302690 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.302696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.304770 (+ 2074us) server_negotiation.cc:407] Connection header received
0619 11:05:28.304950 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.304954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.305003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.305056 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.306047 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.306864 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.308201 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.308664 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.308785 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.311009 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.311015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.311022 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.311053 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.312944 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.313453 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.313459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.313464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.313549 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.313875 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.313878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.313880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.314107 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.314231 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.314604 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.314723 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260619 11:05:28.315826 24046 ts_tablet_manager.cc:1918] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:28.316067 24047 tablet_replica.cc:330] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: stopping tablet replica
I20260619 11:05:28.316195 24046 log.cc:1196] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/e092d117c9264c73b6a99d47707a06d3
I20260619 11:05:28.316571 24046 ts_tablet_manager.cc:1939] T e092d117c9264c73b6a99d47707a06d3 P 25b2f6b36d284fb7aab35e486f3b2015: Deleting consensus metadata
I20260619 11:05:28.316886 24047 raft_consensus.cc:2229] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:28.317255 24047 raft_consensus.cc:2258] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:05:28.318339 23878 connection.cc:441] server connection from 127.0.0.1:55330 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55330 (request call id 1) could send its response
I20260619 11:05:28.319869 24047 ts_tablet_manager.cc:1905] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:28.322391 24047 ts_tablet_manager.cc:1918] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:28.322535 24047 log.cc:1196] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/e092d117c9264c73b6a99d47707a06d3
I20260619 11:05:28.322898 24047 ts_tablet_manager.cc:1939] T e092d117c9264c73b6a99d47707a06d3 P c6cd46533d1e402fb5661105f73741d6: Deleting consensus metadata
W20260619 11:05:28.323851 23741 connection.cc:441] server connection from 127.0.0.1:41326 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41326 (request call id 1) could send its response
I20260619 11:05:28.326089 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.302558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.16.61.60:37719)
0619 11:05:28.302772 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.302777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.316196 (+ 13419us) server_negotiation.cc:407] Connection header received
0619 11:05:28.316400 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.316404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.316458 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.316511 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.317420 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.318175 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.319507 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.319870 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.319941 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.321532 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.321557 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.321560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.321585 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.324121 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.324701 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.324705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.324706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.324789 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.325096 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.325099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.325101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.325313 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.325424 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.325800 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.325941 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260619 11:05:28.326838 24006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.316318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.16.61.61:44913)
0619 11:05:28.316420 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.316424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.317799 (+ 1375us) server_negotiation.cc:407] Connection header received
0619 11:05:28.317944 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.317947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.317983 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.318026 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.318957 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.319682 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.320850 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.321227 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.321298 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.323129 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.323133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.323136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.323162 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.324880 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.325434 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.325440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.325445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.325522 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.325872 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.325878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.325882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.326089 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.326186 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.326572 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.326672 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:05:28.332437 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:28.317344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.16.61.60:37719)
0619 11:05:28.317473 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:28.317478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:28.324438 (+ 6960us) server_negotiation.cc:407] Connection header received
0619 11:05:28.324623 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:28.324626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:28.324663 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:28.324708 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:28.325465 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.326242 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.327582 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:28.327912 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:28.327989 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:28.329376 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:28.329379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:28.329381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:28.329399 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:28.330790 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.331249 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.331251 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.331252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.331340 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:28.331608 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:28.331610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:28.331611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:28.331884 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:28.332001 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:28.332209 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 11:05:28.332307 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:05:29.287266 24057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.286790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.3:47235)
0619 11:05:29.287041 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.287134 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
W20260619 11:05:29.287612 23991 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:45373 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
I20260619 11:05:29.292785 24058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.292353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.1:42439)
0619 11:05:29.292593 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.292677 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":58,"threads_started":1}
W20260619 11:05:29.293049 23718 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:45373 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
I20260619 11:05:29.293288 24059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.292909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.2:44161)
0619 11:05:29.293131 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.293215 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
W20260619 11:05:29.293596 23855 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:45373 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
I20260619 11:05:29.800191 24060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ca84261526b4b5da2e43f353fd49c8f)
I20260619 11:05:29.800393 24060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } }
I20260619 11:05:29.801342 24060 leader_election.cc:288] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ca84261526b4b5da2e43f353fd49c8f (127.16.61.62:45373), 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60:37719)
I20260619 11:05:29.801883 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.801496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:56030)
0619 11:05:29.801683 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.801760 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
W20260619 11:05:29.802105 23338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
I20260619 11:05:29.802924 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.802682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:56036)
0619 11:05:29.802800 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.802850 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 11:05:29.802979 24065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ca84261526b4b5da2e43f353fd49c8f)
W20260619 11:05:29.803040 23338 leader_election.cc:334] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ca84261526b4b5da2e43f353fd49c8f (127.16.61.62:45373): Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
I20260619 11:05:29.803134 24065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } }
I20260619 11:05:29.803995 24065 leader_election.cc:288] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ca84261526b4b5da2e43f353fd49c8f (127.16.61.62:45373), 19087d9aa3d849c1bad2e98b83a56c72 (127.16.61.61:44913)
I20260619 11:05:29.804579 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.804082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:56042)
0619 11:05:29.804400 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.804468 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":60,"threads_started":1}
W20260619 11:05:29.804725 23522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
I20260619 11:05:29.805564 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.805357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:56044)
0619 11:05:29.805461 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.805507 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 11:05:29.805698 23522 leader_election.cc:334] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ca84261526b4b5da2e43f353fd49c8f (127.16.61.62:45373): Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
I20260619 11:05:29.805977 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.801791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:37719 (local address 127.0.0.1:45324)
0619 11:05:29.802021 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.802036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:29.802129 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:29.802525 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:29.802528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:29.802548 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:29.802644 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.802649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.803619 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.803622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.804847 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.804856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.805590 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.805593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.805679 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.805705 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:05:29.805789 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 11:05:29.805854 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":62,"threads_started":1}
I20260619 11:05:29.806083 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.801946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.16.61.60:37719)
0619 11:05:29.802318 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:29.802321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:29.802332 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:05:29.802379 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:29.802381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:29.802418 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:29.802486 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:29.802769 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.803519 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.804979 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.805483 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.805590 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.805649 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:29.805857 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 11:05:29.805963 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":309,"thread_start_us":72,"threads_started":1}
I20260619 11:05:29.806411 23547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19087d9aa3d849c1bad2e98b83a56c72" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" is_pre_election: true
I20260619 11:05:29.806617 23547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19087d9aa3d849c1bad2e98b83a56c72 in term 1.
I20260619 11:05:29.806943 23338 leader_election.cc:302] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19087d9aa3d849c1bad2e98b83a56c72, 6bc8c885cc324a96aea7d4153b21a2ff; no voters: 9ca84261526b4b5da2e43f353fd49c8f
I20260619 11:05:29.807289 24060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 11:05:29.807339 24060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ca84261526b4b5da2e43f353fd49c8f)
I20260619 11:05:29.807366 24060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:05:29.808169 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.804215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.16.61.61:44913)
0619 11:05:29.804481 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:29.804490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:29.804503 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:29.804611 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:29.804615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:29.804660 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:29.804744 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:29.804936 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.805599 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.807131 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.807738 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.807823 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.807863 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:29.808040 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 11:05:29.808086 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260619 11:05:29.808202 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.804154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44913 (local address 127.0.0.1:49282)
0619 11:05:29.804424 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.804437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:29.804515 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:29.804751 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:29.804754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:29.804770 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:29.804855 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.804861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.805796 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.805799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.807035 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.807044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.807843 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.807847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.807926 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.807948 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:05:29.808030 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 11:05:29.808091 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":54,"threads_started":1}
I20260619 11:05:29.808591 23364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "19087d9aa3d849c1bad2e98b83a56c72" is_pre_election: true
I20260619 11:05:29.808799 24060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } }
I20260619 11:05:29.809028 23364 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bc8c885cc324a96aea7d4153b21a2ff in current term 2: Already voted for candidate 19087d9aa3d849c1bad2e98b83a56c72 in this term.
I20260619 11:05:29.809161 24060 leader_election.cc:288] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [CANDIDATE]: Term 2 election: Requested vote from peers 9ca84261526b4b5da2e43f353fd49c8f (127.16.61.62:45373), 6bc8c885cc324a96aea7d4153b21a2ff (127.16.61.60:37719)
I20260619 11:05:29.809459 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.809234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:56058)
0619 11:05:29.809340 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.809394 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 11:05:29.809559 23524 leader_election.cc:302] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bc8c885cc324a96aea7d4153b21a2ff; no voters: 19087d9aa3d849c1bad2e98b83a56c72, 9ca84261526b4b5da2e43f353fd49c8f
I20260619 11:05:29.809743 24065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:05:29.809935 23547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19087d9aa3d849c1bad2e98b83a56c72" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6bc8c885cc324a96aea7d4153b21a2ff"
I20260619 11:05:29.810655 24065 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:05:29.811124 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.810939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:56060)
0619 11:05:29.811026 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.811082 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 11:05:29.811269 23338 leader_election.cc:334] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ca84261526b4b5da2e43f353fd49c8f (127.16.61.62:45373): Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
I20260619 11:05:29.811703 23547 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19087d9aa3d849c1bad2e98b83a56c72 in term 2.
I20260619 11:05:29.811954 23338 leader_election.cc:302] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19087d9aa3d849c1bad2e98b83a56c72, 6bc8c885cc324a96aea7d4153b21a2ff; no voters: 9ca84261526b4b5da2e43f353fd49c8f
I20260619 11:05:29.812112 24060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:05:29.812301 24060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [term 2 LEADER]: Becoming Leader. State: Replica: 19087d9aa3d849c1bad2e98b83a56c72, State: Running, Role: LEADER
I20260619 11:05:29.812523 24060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } }
I20260619 11:05:29.813391 24070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19087d9aa3d849c1bad2e98b83a56c72. Latest consensus state: current_term: 2 leader_uuid: "19087d9aa3d849c1bad2e98b83a56c72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:29.813503 24070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:29.813755 24072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:29.815914 24072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:29.816274 24072 catalog_manager.cc:1259] Loaded cluster ID: ed7a528273cd4b0e842c3ac9f472b358
I20260619 11:05:29.816335 24072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:29.816815 24072 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:29.817176 24072 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Loaded TSK: 0
I20260619 11:05:29.817761 24072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:05:29.836032 23353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:29.838779 23353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:29.841249 23547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [term 2 FOLLOWER]: Refusing update from remote peer 19087d9aa3d849c1bad2e98b83a56c72: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 11:05:29.841302 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.841078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:56068)
0619 11:05:29.841177 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.841224 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 11:05:29.841734 24060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 11:05:29.842042 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.841832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.0.0.1:56078)
0619 11:05:29.841942 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.841995 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:05:29.842159 23338 consensus_peers.cc:487] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 -> Peer 9ca84261526b4b5da2e43f353fd49c8f (127.16.61.62:45373): Couldn't send request to peer 9ca84261526b4b5da2e43f353fd49c8f. Status: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 11:05:29.843036 24065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19087d9aa3d849c1bad2e98b83a56c72. Latest consensus state: current_term: 2 leader_uuid: "19087d9aa3d849c1bad2e98b83a56c72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:29.843138 24065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:05:29.844219 24060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "19087d9aa3d849c1bad2e98b83a56c72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:29.844358 24060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:29.844506 24065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "19087d9aa3d849c1bad2e98b83a56c72" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9ca84261526b4b5da2e43f353fd49c8f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45373 } } peers { permanent_uuid: "19087d9aa3d849c1bad2e98b83a56c72" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44913 } attrs { promote: false } } peers { permanent_uuid: "6bc8c885cc324a96aea7d4153b21a2ff" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 37719 } attrs { promote: false } } }
I20260619 11:05:29.844609 24065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc8c885cc324a96aea7d4153b21a2ff [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:05:29.853925 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.849578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42999 (local address 127.0.0.1:34436)
0619 11:05:29.849678 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.849690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:29.849768 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:29.850492 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:29.850495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:29.850512 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:29.850609 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.850617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.851534 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.851536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.852692 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.852700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.853576 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.853580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.853662 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.853700 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 11:05:29.853747 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 11:05:29.853812 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 11:05:29.854231 24061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.849823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:36265 (local address 127.0.0.1:41340)
0619 11:05:29.850055 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.850067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:29.850140 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:29.850729 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:29.850733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:29.850749 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:29.850832 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.850837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.851797 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.851800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.852913 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.852920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.853863 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.853866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.853950 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.853974 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:05:29.854055 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:05:29.854126 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":23}
I20260619 11:05:29.854554 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.850098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:32829 (local address 127.0.0.1:55346)
0619 11:05:29.850741 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.850753 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:29.850837 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:29.851337 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:29.851340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:29.851356 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:29.851428 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.851435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.852233 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.852239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.853453 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:29.853461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.854208 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.854211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:29.854297 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.854328 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 11:05:29.854395 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 11:05:29.854465 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260619 11:05:29.854566 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.849702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.16.61.1:42999)
0619 11:05:29.850235 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:29.850240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:29.850253 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:29.850316 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:29.850319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:29.850367 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:29.850721 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:29.850838 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.851415 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.852829 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.853453 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.854175 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.854229 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:29.854276 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 11:05:29.854436 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":79,"threads_started":1}
I20260619 11:05:29.854558 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.850291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.16.61.2:36265)
0619 11:05:29.850525 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:29.850530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:29.850543 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:29.850594 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:29.850596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:29.850631 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:29.850723 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:29.850964 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.851705 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.853032 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.853642 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.853736 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.853812 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:29.854369 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 11:05:29.854439 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
I20260619 11:05:29.854949 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.850403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.16.61.3:32829)
0619 11:05:29.851110 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:29.851114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:29.851128 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:05:29.851187 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:29.851189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:29.851231 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:29.851314 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:29.851534 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.852132 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.853565 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:29.853985 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:29.854103 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:29.854157 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:29.854718 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 11:05:29.854807 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":60,"threads_started":1}
I20260619 11:05:29.855005 23788 tablet_service.cc:1467] Processing CreateTablet for tablet 6819f40cee1848e790bd66747179986d (DEFAULT_TABLE table=test-table [id=961c826d0fa24b1e8bf114385bd436d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:29.855091 23651 tablet_service.cc:1467] Processing CreateTablet for tablet 6819f40cee1848e790bd66747179986d (DEFAULT_TABLE table=test-table [id=961c826d0fa24b1e8bf114385bd436d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:29.855211 23788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6819f40cee1848e790bd66747179986d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:29.855324 23651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6819f40cee1848e790bd66747179986d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:29.856092 23924 tablet_service.cc:1467] Processing CreateTablet for tablet 6819f40cee1848e790bd66747179986d (DEFAULT_TABLE table=test-table [id=961c826d0fa24b1e8bf114385bd436d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:29.856346 23924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6819f40cee1848e790bd66747179986d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:29.857388 24083 tablet_bootstrap.cc:492] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9: Bootstrap starting.
I20260619 11:05:29.857858 24084 tablet_bootstrap.cc:492] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: Bootstrap starting.
I20260619 11:05:29.858245 24083 tablet_bootstrap.cc:654] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:29.858511 24085 tablet_bootstrap.cc:492] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: Bootstrap starting.
I20260619 11:05:29.858793 24084 tablet_bootstrap.cc:654] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:29.859442 24085 tablet_bootstrap.cc:654] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:29.859710 24083 tablet_bootstrap.cc:492] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9: No bootstrap required, opened a new log
I20260619 11:05:29.859840 24083 ts_tablet_manager.cc:1397] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:05:29.860200 24084 tablet_bootstrap.cc:492] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: No bootstrap required, opened a new log
I20260619 11:05:29.860280 24084 ts_tablet_manager.cc:1397] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:29.860232 24083 raft_consensus.cc:348] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:29.860396 24083 raft_consensus.cc:374] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:29.860451 24083 raft_consensus.cc:729] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069869d675a740a09d0f034a0566ddd9, State: Initialized, Role: FOLLOWER
I20260619 11:05:29.860577 24083 consensus_queue.cc:260] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:29.860662 24084 raft_consensus.cc:348] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:29.860785 24084 raft_consensus.cc:374] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:29.860826 24084 raft_consensus.cc:729] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cd46533d1e402fb5661105f73741d6, State: Initialized, Role: FOLLOWER
I20260619 11:05:29.860888 24083 ts_tablet_manager.cc:1428] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:29.860967 24084 consensus_queue.cc:260] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:29.861291 24084 ts_tablet_manager.cc:1428] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:29.861572 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.861217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.1:39177)
0619 11:05:29.861423 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.861481 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260619 11:05:29.862354 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.861922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.1:35397)
0619 11:05:29.862240 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.862288 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":80,"threads_started":1}
I20260619 11:05:29.863101 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.862706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.2:50563)
0619 11:05:29.862958 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.863013 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
I20260619 11:05:29.863343 23719 heartbeater.cc:503] Master 127.16.61.61:44913 was elected leader, sending a full tablet report...
I20260619 11:05:29.864162 23856 heartbeater.cc:503] Master 127.16.61.61:44913 was elected leader, sending a full tablet report...
I20260619 11:05:29.864315 24085 tablet_bootstrap.cc:492] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: No bootstrap required, opened a new log
I20260619 11:05:29.864424 24085 ts_tablet_manager.cc:1397] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260619 11:05:29.864815 24085 raft_consensus.cc:348] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:29.864936 24085 raft_consensus.cc:374] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:29.864979 24085 raft_consensus.cc:729] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b2f6b36d284fb7aab35e486f3b2015, State: Initialized, Role: FOLLOWER
I20260619 11:05:29.865113 24085 consensus_queue.cc:260] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:29.865449 24085 ts_tablet_manager.cc:1428] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:05:29.865530 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.864795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.2:50431)
0619 11:05:29.865420 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.865469 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562}
I20260619 11:05:29.866149 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.865809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.3:39567)
0619 11:05:29.866013 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.866066 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260619 11:05:29.866860 23992 heartbeater.cc:503] Master 127.16.61.61:44913 was elected leader, sending a full tablet report...
I20260619 11:05:29.867133 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:29.866920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.3:34075)
0619 11:05:29.867027 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:29.867076 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:05:29.916955 23857 tablet.cc:2369] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:30.108186 23993 tablet.cc:2369] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:30.212771 24089 raft_consensus.cc:482] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:30.212889 24089 raft_consensus.cc:504] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:30.214015 24089 leader_election.cc:288] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6cd46533d1e402fb5661105f73741d6 (127.16.61.2:36265), 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829)
I20260619 11:05:30.217078 24090 raft_consensus.cc:482] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:30.217188 24090 raft_consensus.cc:504] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:30.217592 24090 leader_election.cc:288] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999), 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829)
I20260619 11:05:30.217931 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.214372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:53281 (local address 127.16.61.2:36265)
0619 11:05:30.214497 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:30.214504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:30.214612 (+ 108us) server_negotiation.cc:407] Connection header received
0619 11:05:30.214713 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:30.214715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:30.214797 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:30.214874 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:30.215201 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.215847 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.216842 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.217342 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.217447 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:30.217509 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:30.217733 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 11:05:30.217817 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:05:30.217931 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819f40cee1848e790bd66747179986d" candidate_uuid: "c6cd46533d1e402fb5661105f73741d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b2f6b36d284fb7aab35e486f3b2015" is_pre_election: true
I20260619 11:05:30.217983 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.214281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:32829 (local address 127.16.61.1:59551)
0619 11:05:30.214398 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.214414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:30.214537 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:30.214790 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:30.214793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:30.214808 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:30.214946 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:30.214954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.215818 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.215821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:30.216661 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:30.216670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.217619 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.217625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:30.217715 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:30.217732 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:05:30.217793 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 11:05:30.217844 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 11:05:30.218063 23944 raft_consensus.cc:2454] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cd46533d1e402fb5661105f73741d6 in term 0.
I20260619 11:05:30.218111 23671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819f40cee1848e790bd66747179986d" candidate_uuid: "c6cd46533d1e402fb5661105f73741d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069869d675a740a09d0f034a0566ddd9" is_pre_election: true
I20260619 11:05:30.218220 23671 raft_consensus.cc:2454] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cd46533d1e402fb5661105f73741d6 in term 0.
I20260619 11:05:30.218318 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.214440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:59551 (local address 127.16.61.3:32829)
0619 11:05:30.214560 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:30.214566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:30.214584 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:05:30.214635 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:30.214638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:30.214693 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:30.214764 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:30.215202 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.215715 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.216793 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.217395 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.217508 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:30.217568 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:30.218126 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 11:05:30.218217 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:05:30.218427 23742 leader_election.cc:302] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25b2f6b36d284fb7aab35e486f3b2015, c6cd46533d1e402fb5661105f73741d6; no voters:
I20260619 11:05:30.218657 24090 raft_consensus.cc:2781] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:30.218683 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.214288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:36265 (local address 127.16.61.1:53281)
0619 11:05:30.214522 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.214535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:30.214634 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:30.214893 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:30.214896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:30.214951 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:30.215052 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:30.215056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.215958 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.215960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:30.216700 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:30.216707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.217449 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.217456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:30.217563 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:30.217594 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 11:05:30.217662 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 11:05:30.217729 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":99}
I20260619 11:05:30.218725 24090 raft_consensus.cc:482] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:30.218679 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819f40cee1848e790bd66747179986d" candidate_uuid: "069869d675a740a09d0f034a0566ddd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b2f6b36d284fb7aab35e486f3b2015" is_pre_election: true
I20260619 11:05:30.218767 24090 raft_consensus.cc:3037] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:30.218825 23944 raft_consensus.cc:2454] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 069869d675a740a09d0f034a0566ddd9 in term 0.
I20260619 11:05:30.219137 23605 leader_election.cc:302] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 069869d675a740a09d0f034a0566ddd9, 25b2f6b36d284fb7aab35e486f3b2015; no voters:
I20260619 11:05:30.219163 23808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819f40cee1848e790bd66747179986d" candidate_uuid: "069869d675a740a09d0f034a0566ddd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6cd46533d1e402fb5661105f73741d6" is_pre_election: true
I20260619 11:05:30.219374 24089 raft_consensus.cc:2781] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:30.219434 24089 raft_consensus.cc:482] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:30.219481 24089 raft_consensus.cc:3037] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:30.219998 24090 raft_consensus.cc:504] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:30.220234 23808 raft_consensus.cc:2379] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 069869d675a740a09d0f034a0566ddd9 in current term 1: Already voted for candidate c6cd46533d1e402fb5661105f73741d6 in this term.
I20260619 11:05:30.220335 24090 leader_election.cc:288] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [CANDIDATE]: Term 1 election: Requested vote from peers 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999), 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829)
I20260619 11:05:30.220597 24089 raft_consensus.cc:504] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:30.220719 23944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819f40cee1848e790bd66747179986d" candidate_uuid: "c6cd46533d1e402fb5661105f73741d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b2f6b36d284fb7aab35e486f3b2015"
I20260619 11:05:30.220810 23944 raft_consensus.cc:3037] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:30.220857 23671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819f40cee1848e790bd66747179986d" candidate_uuid: "c6cd46533d1e402fb5661105f73741d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "069869d675a740a09d0f034a0566ddd9"
I20260619 11:05:30.220984 24089 leader_election.cc:288] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [CANDIDATE]: Term 1 election: Requested vote from peers c6cd46533d1e402fb5661105f73741d6 (127.16.61.2:36265), 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829)
I20260619 11:05:30.221005 23671 raft_consensus.cc:2379] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c6cd46533d1e402fb5661105f73741d6 in current term 1: Already voted for candidate 069869d675a740a09d0f034a0566ddd9 in this term.
I20260619 11:05:30.221190 23808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819f40cee1848e790bd66747179986d" candidate_uuid: "069869d675a740a09d0f034a0566ddd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6cd46533d1e402fb5661105f73741d6"
I20260619 11:05:30.221316 23808 raft_consensus.cc:2379] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 069869d675a740a09d0f034a0566ddd9 in current term 1: Already voted for candidate c6cd46533d1e402fb5661105f73741d6 in this term.
I20260619 11:05:30.221524 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6819f40cee1848e790bd66747179986d" candidate_uuid: "069869d675a740a09d0f034a0566ddd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25b2f6b36d284fb7aab35e486f3b2015"
I20260619 11:05:30.221887 23605 leader_election.cc:302] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 069869d675a740a09d0f034a0566ddd9; no voters: 25b2f6b36d284fb7aab35e486f3b2015, c6cd46533d1e402fb5661105f73741d6
I20260619 11:05:30.221911 23944 raft_consensus.cc:2454] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cd46533d1e402fb5661105f73741d6 in term 1.
I20260619 11:05:30.222029 24089 raft_consensus.cc:2726] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 11:05:30.222209 23742 leader_election.cc:302] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25b2f6b36d284fb7aab35e486f3b2015, c6cd46533d1e402fb5661105f73741d6; no voters: 069869d675a740a09d0f034a0566ddd9
I20260619 11:05:30.222397 24090 raft_consensus.cc:2781] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:30.222608 24090 raft_consensus.cc:686] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 1 LEADER]: Becoming Leader. State: Replica: c6cd46533d1e402fb5661105f73741d6, State: Running, Role: LEADER
I20260619 11:05:30.222957 24090 consensus_queue.cc:237] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } }
I20260619 11:05:30.223363 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.223127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.2:37289)
0619 11:05:30.223233 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.223295 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 11:05:30.224666 23354 catalog_manager.cc:5495] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6cd46533d1e402fb5661105f73741d6 (127.16.61.2). New cstate: current_term: 1 leader_uuid: "c6cd46533d1e402fb5661105f73741d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6cd46533d1e402fb5661105f73741d6" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 36265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:30.302162 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.299179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.16.61.2:36265)
0619 11:05:30.299317 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:30.299322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:30.299341 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:05:30.299384 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:30.299386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:30.299425 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:30.299502 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:30.299781 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.300314 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.301133 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:30.301367 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:30.301468 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:30.301696 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:30.301758 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:30.301856 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:30.301975 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 11:05:30.302023 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:05:30.304121 23944 raft_consensus.cc:1261] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 1 FOLLOWER]: Refusing update from remote peer c6cd46533d1e402fb5661105f73741d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:30.304127 23671 raft_consensus.cc:1261] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Refusing update from remote peer c6cd46533d1e402fb5661105f73741d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:30.304509 24096 consensus_queue.cc:1035] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25b2f6b36d284fb7aab35e486f3b2015" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:30.304672 24090 consensus_queue.cc:1035] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "069869d675a740a09d0f034a0566ddd9" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:30.306521 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.306258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.3:38335)
0619 11:05:30.306375 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.306455 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 11:05:30.306586 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.306318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.1:35439)
0619 11:05:30.306463 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.306518 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 11:05:30.310439 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.309873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.3:34593)
0619 11:05:30.310316 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.310369 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
I20260619 11:05:30.310600 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.310311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.1:49989)
0619 11:05:30.310482 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.310537 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 11:05:30.311556 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.311327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.2:43215)
0619 11:05:30.311438 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.311491 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 11:05:30.312119 24100 mvcc.cc:205] Tried to move back new op lower bound from 7298527765721645056 to 7298527765394669568. Current Snapshot: MvccSnapshot[applied={T|T < 7298527765721645056}]
I20260619 11:05:30.313434 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.313219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:45373 (local address 127.16.61.2:51681)
0619 11:05:30.313325 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.313375 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 11:05:30.313755 23353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:30.313896 23353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:30.317394 23353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19087d9aa3d849c1bad2e98b83a56c72: Sending DeleteTablet for 3 replicas of tablet 6819f40cee1848e790bd66747179986d
I20260619 11:05:30.317988 23788 tablet_service.cc:1514] Processing DeleteTablet for tablet 6819f40cee1848e790bd66747179986d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:30 UTC) from {username='oryx', principal='oryx/127.16.61.61@KRBTEST.COM'} at 127.0.0.1:41340
I20260619 11:05:30.317988 23650 tablet_service.cc:1514] Processing DeleteTablet for tablet 6819f40cee1848e790bd66747179986d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:30 UTC) from {username='oryx', principal='oryx/127.16.61.61@KRBTEST.COM'} at 127.0.0.1:34436
I20260619 11:05:30.317999 23924 tablet_service.cc:1514] Processing DeleteTablet for tablet 6819f40cee1848e790bd66747179986d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:30 UTC) from {username='oryx', principal='oryx/127.16.61.61@KRBTEST.COM'} at 127.0.0.1:55346
I20260619 11:05:30.317968 23651 tablet_service.cc:1514] Processing DeleteTablet for tablet 6819f40cee1848e790bd66747179986d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:30 UTC) from {username='oryx', principal='oryx/127.16.61.61@KRBTEST.COM'} at 127.0.0.1:34436
I20260619 11:05:30.318308 24108 tablet_replica.cc:330] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9: stopping tablet replica
I20260619 11:05:30.318432 24108 raft_consensus.cc:2229] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:30.318559 24108 raft_consensus.cc:2258] T 6819f40cee1848e790bd66747179986d P 069869d675a740a09d0f034a0566ddd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:30.318579 24107 tablet_replica.cc:330] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: stopping tablet replica
I20260619 11:05:30.318681 24107 raft_consensus.cc:2229] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:30.318764 24109 tablet_replica.cc:330] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: stopping tablet replica
I20260619 11:05:30.318874 24109 raft_consensus.cc:2229] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:30.318943 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 23498
I20260619 11:05:30.319000 24109 raft_consensus.cc:2258] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:30.318847 24107 raft_consensus.cc:2258] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:30.320655 24107 ts_tablet_manager.cc:1905] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:30.321360 24109 ts_tablet_manager.cc:1905] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:30.323221 24107 ts_tablet_manager.cc:1918] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:30.323290 24107 log.cc:1196] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/6819f40cee1848e790bd66747179986d
I20260619 11:05:30.323572 24107 ts_tablet_manager.cc:1939] T 6819f40cee1848e790bd66747179986d P c6cd46533d1e402fb5661105f73741d6: Deleting consensus metadata
I20260619 11:05:30.324064 24109 ts_tablet_manager.cc:1918] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:30.324141 24109 log.cc:1196] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/6819f40cee1848e790bd66747179986d
I20260619 11:05:30.324261 23341 catalog_manager.cc:4841] TS c6cd46533d1e402fb5661105f73741d6 (127.16.61.2:36265): tablet 6819f40cee1848e790bd66747179986d (table test-table [id=961c826d0fa24b1e8bf114385bd436d7]) successfully deleted
I20260619 11:05:30.324383 24109 ts_tablet_manager.cc:1939] T 6819f40cee1848e790bd66747179986d P 25b2f6b36d284fb7aab35e486f3b2015: Deleting consensus metadata
I20260619 11:05:30.324895 23339 catalog_manager.cc:4841] TS 25b2f6b36d284fb7aab35e486f3b2015 (127.16.61.3:32829): tablet 6819f40cee1848e790bd66747179986d (table test-table [id=961c826d0fa24b1e8bf114385bd436d7]) successfully deleted
I20260619 11:05:30.325466 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 23725
I20260619 11:05:30.325558 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:30.325324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42999 (local address 127.0.0.1:34440)
0619 11:05:30.325418 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:30.325467 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:42999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 11:05:30.325722 23341 catalog_manager.cc:4584] TS 069869d675a740a09d0f034a0566ddd9 (127.16.61.1:42999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6819f40cee1848e790bd66747179986d: Network error: Client connection negotiation failed: client connection to 127.16.61.1:42999: connect: Connection refused (error 111)
I20260619 11:05:30.332863 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 23861
I20260619 11:05:30.338178 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 23218
I20260619 11:05:30.344058 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 23326
2026-06-19T11:05:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5482 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:30 dist-test-slave-8vcz krb5kdc[24112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:30 dist-test-slave-8vcz krb5kdc[24112](info): set up 2 sockets
Jun 19 11:05:30 dist-test-slave-8vcz krb5kdc[24112](info): commencing operation
krb5kdc: starting...
W20260619 11:05:31.388334 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:31 dist-test-slave-8vcz krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.62.keytab.
Entry for principal oryx/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.62.keytab.
I20260619 11:05:31.502601 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:46037
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.62.keytab
--principal=oryx/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:46037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:31.605141 24125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:31.605394 24125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:31.605460 24125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:31.609395 24125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:31.609467 24125 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 11:05:31.609499 24125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:31.609555 24125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:31.609594 24125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:31.609630 24125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:31.614645 24125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:46037
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.62.keytab
--principal=oryx/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:46037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:31.615549 24125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:31.616489 24125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:31.624174 24130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:31.624172 24131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:31.624187 24133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:31.624604 24125 server_base.cc:1034] running on GCE node
I20260619 11:05:31.625126 24125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:31.626114 24125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:31.627288 24125 hybrid_clock.cc:648] HybridClock initialized: now 1781867131627272 us; error 30 us; skew 500 ppm
Jun 19 11:05:31 dist-test-slave-8vcz krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867131, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:31.629895 24125 init.cc:377] Logged in from keytab as oryx/127.16.61.62@KRBTEST.COM (short username oryx)
I20260619 11:05:31.630721 24125 webserver.cc:460] Webserver started at http://127.16.61.62:43793/ using document root <none> and password file <none>
I20260619 11:05:31.631253 24125 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:31.631337 24125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:31.631546 24125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:31.633461 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "08120b984a6a48ae94ad3aa04690799b"
format_stamp: "Formatted at 2026-06-19 11:05:31 on dist-test-slave-8vcz"
server_key: "6296701b0cfb754531793da1b2647ace"
server_key_iv: "fa46025eeea4932f37ee0a6daf0d6476"
server_key_version: "encryptionkey@0"
I20260619 11:05:31.633992 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "08120b984a6a48ae94ad3aa04690799b"
format_stamp: "Formatted at 2026-06-19 11:05:31 on dist-test-slave-8vcz"
server_key: "6296701b0cfb754531793da1b2647ace"
server_key_iv: "fa46025eeea4932f37ee0a6daf0d6476"
server_key_version: "encryptionkey@0"
I20260619 11:05:31.637424 24125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:31.639546 24140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:31.640700 24125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:31.640827 24125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "08120b984a6a48ae94ad3aa04690799b"
format_stamp: "Formatted at 2026-06-19 11:05:31 on dist-test-slave-8vcz"
server_key: "6296701b0cfb754531793da1b2647ace"
server_key_iv: "fa46025eeea4932f37ee0a6daf0d6476"
server_key_version: "encryptionkey@0"
I20260619 11:05:31.640925 24125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:31.679576 24125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:31.682633 24125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:31.682821 24125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:31.690245 24125 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:46037
I20260619 11:05:31.690270 24202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:46037 every 8 connection(s)
I20260619 11:05:31.691107 24125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:31.693722 24203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:31.697937 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 24125
I20260619 11:05:31.698086 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:31.698405 16628 external_mini_cluster.cc:1428] Setting key 48bc5a3126d15f6f1b53178b984e50e4
I20260619 11:05:31.699121 24203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: Bootstrap starting.
I20260619 11:05:31.701354 24203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:31.702055 24203 log.cc:824] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:31.703776 24203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: No bootstrap required, opened a new log
Jun 19 11:05:31 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:31.707234 24203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08120b984a6a48ae94ad3aa04690799b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46037 } }
I20260619 11:05:31.707459 24203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:31.707533 24203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08120b984a6a48ae94ad3aa04690799b, State: Initialized, Role: FOLLOWER
I20260619 11:05:31.708076 24203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08120b984a6a48ae94ad3aa04690799b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46037 } }
I20260619 11:05:31.708251 24203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:31.708320 24203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:31.708418 24203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:31.709406 24203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08120b984a6a48ae94ad3aa04690799b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46037 } }
I20260619 11:05:31.709741 24203 leader_election.cc:302] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08120b984a6a48ae94ad3aa04690799b; no voters:
I20260619 11:05:31.710057 24203 leader_election.cc:288] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:31.710142 24208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:31.710351 24208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [term 1 LEADER]: Becoming Leader. State: Replica: 08120b984a6a48ae94ad3aa04690799b, State: Running, Role: LEADER
I20260619 11:05:31.710752 24208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08120b984a6a48ae94ad3aa04690799b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46037 } }
I20260619 11:05:31.711251 24203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:31.712415 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08120b984a6a48ae94ad3aa04690799b. Latest consensus state: current_term: 1 leader_uuid: "08120b984a6a48ae94ad3aa04690799b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08120b984a6a48ae94ad3aa04690799b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46037 } } }
I20260619 11:05:31.712529 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:31.712706 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:31.699770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.16.61.62:46037)
0619 11:05:31.700174 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:31.700188 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:31.700220 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:05:31.700811 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:31.700826 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:31.701121 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:31.701438 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:31.702183 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.703107 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.703965 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.704413 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.704568 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:31.707072 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:31.707076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:31.707088 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:31.707111 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:31.709053 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:31.709450 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:31.709456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:31.709461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:31.709593 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:31.709840 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:31.709845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:31.709847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:31.710360 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:31.711134 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:31.711275 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 11:05:31.711705 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260619 11:05:31.712890 24217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:31.712857 24209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08120b984a6a48ae94ad3aa04690799b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08120b984a6a48ae94ad3aa04690799b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46037 } } }
I20260619 11:05:31.712934 24209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:31.715615 24217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:31.720494 24217 catalog_manager.cc:1347] Generated new cluster ID: eae7cc2ef4e0495eaf3810cb9da639e8
I20260619 11:05:31.720558 24217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:31.743098 24217 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:31.744081 24217 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:31.748353 24217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: Generated new TSK 0
I20260619 11:05:31.748906 24217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab.
Entry for principal oryx/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab.
I20260619 11:05:31.776587 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46037
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab
--principal=oryx/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:31.881608 24229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:31.881817 24229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:31.881857 24229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:31.885634 24229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:31.885679 24229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:31.885715 24229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:31.885751 24229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:31.890533 24229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab
--principal=oryx/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:31.891412 24229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:31.892278 24229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:31.900945 24234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:31.901093 24229 server_base.cc:1034] running on GCE node
W20260619 11:05:31.900954 24237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:31.901034 24235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:31.901602 24229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:31.902123 24229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:31.903266 24229 hybrid_clock.cc:648] HybridClock initialized: now 1781867131903254 us; error 24 us; skew 500 ppm
Jun 19 11:05:31 dist-test-slave-8vcz krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867131, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:31.905812 24229 init.cc:377] Logged in from keytab as oryx/127.16.61.1@KRBTEST.COM (short username oryx)
I20260619 11:05:31.906566 24229 webserver.cc:460] Webserver started at http://127.16.61.1:44869/ using document root <none> and password file <none>
I20260619 11:05:31.907075 24229 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:31.907135 24229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:31.907336 24229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:31.909236 24229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "8da958cb952247628042110e3b1104a1"
format_stamp: "Formatted at 2026-06-19 11:05:31 on dist-test-slave-8vcz"
server_key: "831b7dcb0b1455abd6050cc50731c7c8"
server_key_iv: "c705bc4050c60707bf7a135c23f97446"
server_key_version: "encryptionkey@0"
I20260619 11:05:31.909770 24229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "8da958cb952247628042110e3b1104a1"
format_stamp: "Formatted at 2026-06-19 11:05:31 on dist-test-slave-8vcz"
server_key: "831b7dcb0b1455abd6050cc50731c7c8"
server_key_iv: "c705bc4050c60707bf7a135c23f97446"
server_key_version: "encryptionkey@0"
I20260619 11:05:31.913390 24229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 11:05:31.915563 24244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:31.916680 24229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:05:31.916802 24229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "8da958cb952247628042110e3b1104a1"
format_stamp: "Formatted at 2026-06-19 11:05:31 on dist-test-slave-8vcz"
server_key: "831b7dcb0b1455abd6050cc50731c7c8"
server_key_iv: "c705bc4050c60707bf7a135c23f97446"
server_key_version: "encryptionkey@0"
I20260619 11:05:31.916904 24229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:31.934451 24229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:31.937493 24229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:31.937628 24229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:31.938839 24229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:31.938912 24229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:31.938982 24229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:31.939028 24229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:05:31 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867131, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:31.949695 24229 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:33437
I20260619 11:05:31.949832 24359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:33437 every 8 connection(s)
I20260619 11:05:31.950708 24229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:31.951834 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 24229
I20260619 11:05:31.951953 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:31.952045 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:31.939708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:49357 (local address 127.16.61.62:46037)
0619 11:05:31.939829 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:31.939832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:31.940820 (+ 988us) server_negotiation.cc:407] Connection header received
0619 11:05:31.941761 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:31.941768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:31.941834 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:31.941959 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:31.943275 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.944197 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.945506 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.945738 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.945856 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:31.948205 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:31.948208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:31.948210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:31.948232 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:31.949667 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:31.950233 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:31.950238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:31.950240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:31.950323 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:31.950639 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:31.950644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:31.950646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:31.950865 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:31.950967 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:31.951793 (+ 826us) server_negotiation.cc:299] Negotiation successful
0619 11:05:31.951917 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:05:31.952319 16628 external_mini_cluster.cc:1428] Setting key a93157e1213e7f81fc2f26ef2d1bede2
I20260619 11:05:31.952725 24253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:31.940215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46037 (local address 127.16.61.1:49357)
0619 11:05:31.940666 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:31.940705 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:31.941533 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:31.942100 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:31.942110 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:31.942485 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:31.943088 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:31.943110 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.944332 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.944335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:31.945389 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:31.945398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.945898 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.945902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:31.946061 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:31.946585 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:31.946608 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:31.948041 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:31.949813 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:31.949818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:31.949832 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:31.950136 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:31.950410 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:31.950415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:31.950417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:31.950548 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:31.950976 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:31.950985 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:31.951250 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 11:05:31.951435 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 11:05:31.951778 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
I20260619 11:05:31.954119 24360 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46037
I20260619 11:05:31.954273 24360 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:31.954720 24360 heartbeater.cc:511] Master 127.16.61.62:46037 requested a full tablet report, sending...
I20260619 11:05:31.956126 24157 ts_manager.cc:194] Registered new tserver with Master: 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437)
I20260619 11:05:31.957314 24157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:49357
WARNING: no policy specified for oryx/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.2@KRBTEST.COM" created.
I20260619 11:05:31.964618 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:31.958034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46037 (local address 127.16.61.1:36017)
0619 11:05:31.958270 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:31.958284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:31.958410 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:31.958633 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:31.958635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:31.958827 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:31.958914 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:31.958919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.959616 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.959619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:31.960203 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:31.960211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.960640 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.960643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:31.960731 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:31.961197 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:31.961210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:31.961504 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:31.963203 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:31.963206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:31.963208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:31.963399 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:31.963659 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:31.963682 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:31.963684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:31.963773 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:31.964109 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:31.964112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:31.964195 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 11:05:31.964309 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 11:05:31.964449 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260619 11:05:31.964743 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:31.958068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:36017 (local address 127.16.61.62:46037)
0619 11:05:31.958191 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:31.958196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:31.958362 (+ 166us) server_negotiation.cc:407] Connection header received
0619 11:05:31.958496 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:31.958499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:31.958537 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:31.958621 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:31.959018 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.959504 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.960321 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:31.960539 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:31.960639 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:31.961641 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:31.961645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:31.961647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:31.961668 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:31.963084 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:31.963492 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:31.963497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:31.963499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:31.963574 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:31.963862 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:31.963865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:31.963866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:31.964019 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:31.964119 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:31.964398 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 11:05:31.964532 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab.
Entry for principal oryx/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab.
I20260619 11:05:31.978299 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46037
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab
--principal=oryx/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:32.082482 24369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:32.082746 24369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:32.082815 24369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:32.086772 24369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:32.086854 24369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:32.086927 24369 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:32.087004 24369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:32.092021 24369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab
--principal=oryx/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:32.092935 24369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:32.093842 24369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:32.102586 24377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:32.102613 24375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:32.102610 24374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:32.102931 24369 server_base.cc:1034] running on GCE node
I20260619 11:05:32.103276 24369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:32.103947 24369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:32.105121 24369 hybrid_clock.cc:648] HybridClock initialized: now 1781867132105097 us; error 33 us; skew 500 ppm
Jun 19 11:05:32 dist-test-slave-8vcz krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867132, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:32.111979 24369 init.cc:377] Logged in from keytab as oryx/127.16.61.2@KRBTEST.COM (short username oryx)
I20260619 11:05:32.112851 24369 webserver.cc:460] Webserver started at http://127.16.61.2:39771/ using document root <none> and password file <none>
I20260619 11:05:32.113418 24369 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:32.113503 24369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:32.113696 24369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:32.115598 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d"
format_stamp: "Formatted at 2026-06-19 11:05:32 on dist-test-slave-8vcz"
server_key: "6cb37cc118db824680d68ad56bb4938e"
server_key_iv: "6e8495955439177fa846a2829de6ff3c"
server_key_version: "encryptionkey@0"
I20260619 11:05:32.116194 24369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d"
format_stamp: "Formatted at 2026-06-19 11:05:32 on dist-test-slave-8vcz"
server_key: "6cb37cc118db824680d68ad56bb4938e"
server_key_iv: "6e8495955439177fa846a2829de6ff3c"
server_key_version: "encryptionkey@0"
I20260619 11:05:32.119774 24369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:32.122063 24384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:32.123126 24369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:32.123247 24369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d"
format_stamp: "Formatted at 2026-06-19 11:05:32 on dist-test-slave-8vcz"
server_key: "6cb37cc118db824680d68ad56bb4938e"
server_key_iv: "6e8495955439177fa846a2829de6ff3c"
server_key_version: "encryptionkey@0"
I20260619 11:05:32.123347 24369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:32.138464 24369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:32.141127 24369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:32.141309 24369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:32.142619 24369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:32.142686 24369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:32.142761 24369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:32.142802 24369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 11:05:32 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867132, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:32.153484 24369 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:37269
I20260619 11:05:32.153569 24499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:37269 every 8 connection(s)
I20260619 11:05:32.154556 24369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:32.156731 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.143318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:52675 (local address 127.16.61.62:46037)
0619 11:05:32.143435 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.143439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.144524 (+ 1085us) server_negotiation.cc:407] Connection header received
0619 11:05:32.145533 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.145539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.145605 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.145706 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:32.146956 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.147873 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.149048 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.149383 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.149493 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.152905 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:32.152908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:32.152910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:32.152932 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:32.154688 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.155275 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.155278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.155280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.155371 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.155788 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.155794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.155799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.156025 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:32.156104 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.156529 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.156619 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:05:32.157485 24393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.143875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46037 (local address 127.16.61.2:52675)
0619 11:05:32.144346 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.144392 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.145287 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.145933 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.145940 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.146316 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:32.146806 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.146819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.147998 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.148001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.148929 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.148938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.149530 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.149534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.149699 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.150245 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:32.150271 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:32.152736 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:32.154802 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:32.154808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:32.154824 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:32.155151 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:32.155468 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:32.155471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:32.155473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:32.155620 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:32.156112 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:32.156116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:32.156304 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.156462 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.156670 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260619 11:05:32.158516 24500 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46037
I20260619 11:05:32.158654 24500 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:32.159090 24500 heartbeater.cc:511] Master 127.16.61.62:46037 requested a full tablet report, sending...
I20260619 11:05:32.160142 24157 ts_manager.cc:194] Registered new tserver with Master: a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269)
I20260619 11:05:32.160784 24157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:52675
I20260619 11:05:32.164016 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 24369
I20260619 11:05:32.164147 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:32.164469 16628 external_mini_cluster.cc:1428] Setting key 469956eb32f1a86caafca0ff419eb9a4
I20260619 11:05:32.167307 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.160542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:36423 (local address 127.16.61.62:46037)
0619 11:05:32.160653 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.160660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.160870 (+ 210us) server_negotiation.cc:407] Connection header received
0619 11:05:32.160964 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.160967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.161010 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.161084 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:32.161526 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.162249 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.162981 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.163176 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.163253 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.164392 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:32.164395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:32.164397 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:32.164416 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:32.165729 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.166162 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.166165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.166166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.166229 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.166512 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.166514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.166515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.166690 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:32.166800 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.167053 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.167152 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:05:32.167315 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.160478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46037 (local address 127.16.61.2:36423)
0619 11:05:32.160780 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.160791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.160876 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.161116 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.161119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.161309 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:32.161414 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.161419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.162355 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.162357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.162888 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.162894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.163288 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.163290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.163358 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.163857 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:32.163871 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:32.164272 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:32.165822 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:32.165826 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:32.165828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:32.166066 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:32.166307 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:32.166313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:32.166318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:32.166415 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:32.166790 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:32.166793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:32.166881 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.166993 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.167122 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for oryx/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab.
Entry for principal oryx/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab.
I20260619 11:05:32.189114 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46037
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab
--principal=oryx/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:32.295509 24509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:32.295775 24509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:32.295820 24509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:32.300005 24509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:32.300051 24509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:32.300088 24509 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 11:05:32.300177 24509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:32.305119 24509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab
--principal=oryx/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.61.62:46037
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:32.305967 24509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:32.306840 24509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:32.315825 24514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:32.315814 24517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:32.315810 24515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:32.315869 24509 server_base.cc:1034] running on GCE node
I20260619 11:05:32.316342 24509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:32.316905 24509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:32.318078 24509 hybrid_clock.cc:648] HybridClock initialized: now 1781867132318064 us; error 27 us; skew 500 ppm
Jun 19 11:05:32 dist-test-slave-8vcz krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867132, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:32.320709 24509 init.cc:377] Logged in from keytab as oryx/127.16.61.3@KRBTEST.COM (short username oryx)
I20260619 11:05:32.321485 24509 webserver.cc:460] Webserver started at http://127.16.61.3:40069/ using document root <none> and password file <none>
I20260619 11:05:32.321988 24509 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:32.322070 24509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:32.322268 24509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:32.324121 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "e8627de73e2946fcaf7edef45a0c1820"
format_stamp: "Formatted at 2026-06-19 11:05:32 on dist-test-slave-8vcz"
server_key: "fefe709253c2e019933ddac2f45cd4ac"
server_key_iv: "94c419db354bce266a9e1cfee3c78de4"
server_key_version: "encryptionkey@0"
I20260619 11:05:32.324648 24509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "e8627de73e2946fcaf7edef45a0c1820"
format_stamp: "Formatted at 2026-06-19 11:05:32 on dist-test-slave-8vcz"
server_key: "fefe709253c2e019933ddac2f45cd4ac"
server_key_iv: "94c419db354bce266a9e1cfee3c78de4"
server_key_version: "encryptionkey@0"
I20260619 11:05:32.327996 24509 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:32.330111 24524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:32.331166 24509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:32.331283 24509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "e8627de73e2946fcaf7edef45a0c1820"
format_stamp: "Formatted at 2026-06-19 11:05:32 on dist-test-slave-8vcz"
server_key: "fefe709253c2e019933ddac2f45cd4ac"
server_key_iv: "94c419db354bce266a9e1cfee3c78de4"
server_key_version: "encryptionkey@0"
I20260619 11:05:32.331382 24509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:32.342818 24509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:32.345538 24509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:32.345713 24509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:32.346894 24509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:32.346979 24509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:32.347044 24509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:32.347082 24509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:32.356292 24509 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:40493
Jun 19 11:05:32 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867132, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:32.356367 24639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:40493 every 8 connection(s)
I20260619 11:05:32.357183 24509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:32.360208 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.347630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:33167 (local address 127.16.61.62:46037)
0619 11:05:32.347779 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.347784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.348557 (+ 773us) server_negotiation.cc:407] Connection header received
0619 11:05:32.349680 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.349687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.349753 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.349854 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:32.351239 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.352166 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.353732 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.354120 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.354237 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.356906 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:32.356910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:32.356913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:32.356939 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:32.358508 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.358959 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.358961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.358962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.359052 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.359322 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.359324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.359325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.359482 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:32.359554 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.360011 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.360096 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 11:05:32.360917 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.348033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46037 (local address 127.16.61.3:33167)
0619 11:05:32.348424 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.348468 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.349431 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.350025 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.350033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.350455 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:32.351070 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.351087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.352300 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.352304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.353614 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.353623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.354274 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.354280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.354508 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.355163 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:32.355183 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:32.356752 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:32.358618 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:32.358622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:32.358631 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:32.358880 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:32.359140 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:32.359142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:32.359143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:32.359255 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:32.359563 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:32.359567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:32.359809 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.359967 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.360171 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260619 11:05:32.361936 24640 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46037
I20260619 11:05:32.362074 24640 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:32.362488 24640 heartbeater.cc:511] Master 127.16.61.62:46037 requested a full tablet report, sending...
I20260619 11:05:32.363436 24157 ts_manager.cc:194] Registered new tserver with Master: e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493)
I20260619 11:05:32.364120 24157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:33167
I20260619 11:05:32.365181 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 24509
I20260619 11:05:32.365298 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:32.365581 16628 external_mini_cluster.cc:1428] Setting key d4d45ab879e8ca33b917f0e8de76fe86
I20260619 11:05:32.367475 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:05:32.370399 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.363830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:54575 (local address 127.16.61.62:46037)
0619 11:05:32.363937 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.363942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.364106 (+ 164us) server_negotiation.cc:407] Connection header received
0619 11:05:32.364214 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.364218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.364261 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.364336 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:32.364709 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.365179 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.366121 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.366322 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.366429 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.367448 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:32.367452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:32.367454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:32.367476 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:32.368986 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.369343 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.369349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.369353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.369446 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.369753 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.369755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.369756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.369898 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:32.369969 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.370233 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.370306 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:05:32.370563 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.363783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46037 (local address 127.16.61.3:54575)
0619 11:05:32.364034 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.364045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.364142 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.364350 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.364352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.364527 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:32.364607 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.364612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.365291 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.365294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.366019 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.366030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.366422 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.366425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.366515 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.366995 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:32.367006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:32.367314 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:32.369083 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:32.369085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:32.369087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:32.369256 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:32.369556 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:32.369560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:32.369563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:32.369661 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:32.369980 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:32.369985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:32.370079 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.370193 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.370329 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":102,"threads_started":1}
Jun 19 11:05:32 dist-test-slave-8vcz krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 11:05:32 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:32 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:05:32.392969 24206 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:32.387007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.16.61.62:46037)
0619 11:05:32.387111 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.387116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.387133 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:05:32.387164 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.387166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.387198 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.387271 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:32.387977 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.388454 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.389239 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.389434 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.389504 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.392818 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47476: BlockingRecv error: recv got EOF from 127.0.0.1:47476 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 11:05:32 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:32.401695 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.394209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.16.61.62:46037)
0619 11:05:32.394333 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.394338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.394461 (+ 123us) server_negotiation.cc:407] Connection header received
0619 11:05:32.394589 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.394592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.394630 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.394707 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:32.395256 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.395784 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.396529 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.396734 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.396833 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.398877 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:32.398880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:32.398882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:32.398901 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:32.400156 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.400560 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.400566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.400567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.400642 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:32.400914 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:32.400917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:32.400919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:32.401077 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:32.401213 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.401444 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.401548 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:05:32.404284 24157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:32.407264 24157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:32.421278 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.416124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40493 (local address 127.0.0.1:52254)
0619 11:05:32.416448 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.416462 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.416547 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.417428 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.417431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.417449 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:32.417594 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.417601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.418915 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.418918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.420013 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.420022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.420767 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.420771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.420875 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.420910 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.421016 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.421129 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260619 11:05:32.421698 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.416502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.16.61.3:40493)
0619 11:05:32.417081 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.417089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.417106 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:05:32.417162 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.417166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.417297 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.417451 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:32.417722 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.418794 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.420110 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.420653 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.420769 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.420933 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.421475 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.421565 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":105,"threads_started":1}
I20260619 11:05:32.422071 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.416657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.16.61.1:33437)
0619 11:05:32.416999 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.417013 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.417032 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:05:32.417090 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.417094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.417242 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.417400 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:32.417695 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.418585 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.420089 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.420894 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.421078 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.421195 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.421816 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.421910 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":81,"threads_started":1}
I20260619 11:05:32.422355 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.415624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33437 (local address 127.0.0.1:51756)
0619 11:05:32.416138 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.416187 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.416303 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.417384 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.417390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.417416 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:32.417588 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.417594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.418709 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.418712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.419956 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.419967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.421243 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.421246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.421343 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.421364 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.421446 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.421522 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":24,"thread_start_us":133,"threads_started":1}
I20260619 11:05:32.422890 24663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.416211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:37269 (local address 127.0.0.1:50922)
0619 11:05:32.416713 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.416726 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.416800 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.418139 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.418142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.418160 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:32.418236 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.418242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.419731 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.419735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.420915 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.420936 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.422490 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.422494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.422590 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.422612 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.422684 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.422759 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":98,"threads_started":1}
I20260619 11:05:32.424019 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.416638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.16.61.2:37269)
0619 11:05:32.417724 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.417735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.417755 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:05:32.417838 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.417842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.418002 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.418142 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:32.418350 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.419588 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.421023 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.422189 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.422281 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.422419 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.423794 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.423891 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":100,"threads_started":1}
I20260619 11:05:32.424724 24574 tablet_service.cc:1467] Processing CreateTablet for tablet cb8956fc97b944b9ad1775ac6345d73f (DEFAULT_TABLE table=test-table [id=707931d1fc8c464fbcb157e0cc94c895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:32.424726 24294 tablet_service.cc:1467] Processing CreateTablet for tablet cb8956fc97b944b9ad1775ac6345d73f (DEFAULT_TABLE table=test-table [id=707931d1fc8c464fbcb157e0cc94c895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:32.425725 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8956fc97b944b9ad1775ac6345d73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:32.425745 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8956fc97b944b9ad1775ac6345d73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:32.426286 24434 tablet_service.cc:1467] Processing CreateTablet for tablet cb8956fc97b944b9ad1775ac6345d73f (DEFAULT_TABLE table=test-table [id=707931d1fc8c464fbcb157e0cc94c895]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:32.427165 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8956fc97b944b9ad1775ac6345d73f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:32.432085 24668 tablet_bootstrap.cc:492] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Bootstrap starting.
I20260619 11:05:32.434110 24669 tablet_bootstrap.cc:492] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Bootstrap starting.
I20260619 11:05:32.434554 24670 tablet_bootstrap.cc:492] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Bootstrap starting.
I20260619 11:05:32.434862 24668 tablet_bootstrap.cc:654] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:32.435811 24668 log.cc:824] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:32.436317 24669 tablet_bootstrap.cc:654] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:32.436661 24670 tablet_bootstrap.cc:654] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:32.437072 24669 log.cc:824] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:32.437469 24670 log.cc:824] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:32.437829 24668 tablet_bootstrap.cc:492] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: No bootstrap required, opened a new log
I20260619 11:05:32.438007 24668 ts_tablet_manager.cc:1397] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:05:32.438860 24669 tablet_bootstrap.cc:492] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: No bootstrap required, opened a new log
I20260619 11:05:32.439095 24669 ts_tablet_manager.cc:1397] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:05:32.439144 24670 tablet_bootstrap.cc:492] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: No bootstrap required, opened a new log
I20260619 11:05:32.439318 24670 ts_tablet_manager.cc:1397] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:05:32.441151 24668 raft_consensus.cc:348] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.441382 24668 raft_consensus.cc:374] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:32.441453 24668 raft_consensus.cc:729] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da958cb952247628042110e3b1104a1, State: Initialized, Role: FOLLOWER
I20260619 11:05:32.441874 24668 consensus_queue.cc:260] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.442451 24360 heartbeater.cc:503] Master 127.16.61.62:46037 was elected leader, sending a full tablet report...
I20260619 11:05:32.442751 24668 ts_tablet_manager.cc:1428] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:05:32.442977 24670 raft_consensus.cc:348] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.443207 24670 raft_consensus.cc:374] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:32.443264 24670 raft_consensus.cc:729] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fe6508b8c7401a93eeaac98bdf2d1d, State: Initialized, Role: FOLLOWER
I20260619 11:05:32.443648 24670 consensus_queue.cc:260] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.443559 24669 raft_consensus.cc:348] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.443820 24669 raft_consensus.cc:374] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:32.443915 24669 raft_consensus.cc:729] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8627de73e2946fcaf7edef45a0c1820, State: Initialized, Role: FOLLOWER
I20260619 11:05:32.444306 24500 heartbeater.cc:503] Master 127.16.61.62:46037 was elected leader, sending a full tablet report...
I20260619 11:05:32.444380 24669 consensus_queue.cc:260] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.444651 24670 ts_tablet_manager.cc:1428] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 11:05:32.445228 24669 ts_tablet_manager.cc:1428] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:05:32.445658 24640 heartbeater.cc:503] Master 127.16.61.62:46037 was elected leader, sending a full tablet report...
W20260619 11:05:32.452257 24361 tablet.cc:2369] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:32.608332 24641 tablet.cc:2369] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:32.655709 24501 tablet.cc:2369] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:32.779328 24676 raft_consensus.cc:482] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:32.779556 24676 raft_consensus.cc:504] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.780637 24676 leader_election.cc:288] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437), a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269)
I20260619 11:05:32.784236 24533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.780877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:37269 (local address 127.16.61.3:41741)
0619 11:05:32.780979 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.780995 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.781083 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.781341 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.781344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.781370 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:32.781498 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.781507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.782262 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.782264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.783298 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.783308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.783938 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.783941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.784013 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.784031 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.784087 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.784140 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 11:05:32.784291 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.780980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:41741 (local address 127.16.61.2:37269)
0619 11:05:32.781112 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.781117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.781133 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:05:32.781186 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.781189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.781241 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.781306 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:32.781620 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.782164 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.783418 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.783810 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.783941 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.784003 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.784118 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.784178 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:05:32.784937 24454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8956fc97b944b9ad1775ac6345d73f" candidate_uuid: "e8627de73e2946fcaf7edef45a0c1820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" is_pre_election: true
I20260619 11:05:32.785010 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.781030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:45941 (local address 127.16.61.1:33437)
0619 11:05:32.781155 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.781161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.781448 (+ 287us) server_negotiation.cc:407] Connection header received
0619 11:05:32.781506 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.781508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.781546 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.781638 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:32.782216 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.782722 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.783917 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.784504 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.784624 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.784681 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.784867 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.784925 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:05:32.785051 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.780878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33437 (local address 127.16.61.3:45941)
0619 11:05:32.781341 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.781354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.781436 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.781657 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.781659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.781674 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:32.782099 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.782105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.782837 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.782840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.783802 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.783809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.784639 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.784642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.784743 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.784765 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.784850 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.784908 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":94,"thread_start_us":111,"threads_started":1}
I20260619 11:05:32.785231 24454 raft_consensus.cc:2454] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8627de73e2946fcaf7edef45a0c1820 in term 0.
I20260619 11:05:32.785677 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8956fc97b944b9ad1775ac6345d73f" candidate_uuid: "e8627de73e2946fcaf7edef45a0c1820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da958cb952247628042110e3b1104a1" is_pre_election: true
I20260619 11:05:32.785742 24527 leader_election.cc:302] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6fe6508b8c7401a93eeaac98bdf2d1d, e8627de73e2946fcaf7edef45a0c1820; no voters:
I20260619 11:05:32.785948 24314 raft_consensus.cc:2454] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8627de73e2946fcaf7edef45a0c1820 in term 0.
I20260619 11:05:32.785976 24676 raft_consensus.cc:2781] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:32.786062 24676 raft_consensus.cc:482] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:32.786114 24676 raft_consensus.cc:3037] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:32.787158 24676 raft_consensus.cc:504] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.787518 24676 leader_election.cc:288] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [CANDIDATE]: Term 1 election: Requested vote from peers 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437), a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269)
I20260619 11:05:32.787848 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8956fc97b944b9ad1775ac6345d73f" candidate_uuid: "e8627de73e2946fcaf7edef45a0c1820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da958cb952247628042110e3b1104a1"
I20260619 11:05:32.787892 24454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8956fc97b944b9ad1775ac6345d73f" candidate_uuid: "e8627de73e2946fcaf7edef45a0c1820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d"
I20260619 11:05:32.787971 24314 raft_consensus.cc:3037] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:32.787997 24454 raft_consensus.cc:3037] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:32.788901 24454 raft_consensus.cc:2454] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8627de73e2946fcaf7edef45a0c1820 in term 1.
I20260619 11:05:32.788971 24314 raft_consensus.cc:2454] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8627de73e2946fcaf7edef45a0c1820 in term 1.
I20260619 11:05:32.789186 24527 leader_election.cc:302] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6fe6508b8c7401a93eeaac98bdf2d1d, e8627de73e2946fcaf7edef45a0c1820; no voters:
I20260619 11:05:32.789376 24676 raft_consensus.cc:2781] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:32.789598 24676 raft_consensus.cc:686] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 1 LEADER]: Becoming Leader. State: Replica: e8627de73e2946fcaf7edef45a0c1820, State: Running, Role: LEADER
I20260619 11:05:32.789916 24676 consensus_queue.cc:237] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } }
I20260619 11:05:32.792840 24157 catalog_manager.cc:5495] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3). New cstate: current_term: 1 leader_uuid: "e8627de73e2946fcaf7edef45a0c1820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:32.869902 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.866411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.16.61.3:40493)
0619 11:05:32.866565 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.866570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.866585 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:05:32.866716 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.866718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.866770 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.866877 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:32.867127 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.867641 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.868519 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.868739 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.868845 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.869106 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:32.869518 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:32.869636 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.869709 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.869761 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:05:32.875104 24314 raft_consensus.cc:1261] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Refusing update from remote peer e8627de73e2946fcaf7edef45a0c1820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:32.875105 24454 raft_consensus.cc:1261] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Refusing update from remote peer e8627de73e2946fcaf7edef45a0c1820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:32.875727 24676 consensus_queue.cc:1035] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:32.875900 24678 consensus_queue.cc:1035] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:32.884104 24682 mvcc.cc:205] Tried to move back new op lower bound from 7298527776250351616 to 7298527775909859328. Current Snapshot: MvccSnapshot[applied={T|T < 7298527776250351616}]
I20260619 11:05:32.889469 24155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:32.889591 24155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:32.891707 24155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: Sending DeleteTablet for 3 replicas of tablet cb8956fc97b944b9ad1775ac6345d73f
I20260619 11:05:32.892373 24294 tablet_service.cc:1514] Processing DeleteTablet for tablet cb8956fc97b944b9ad1775ac6345d73f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:32 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:51756
I20260619 11:05:32.892457 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet cb8956fc97b944b9ad1775ac6345d73f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:32 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:52254
I20260619 11:05:32.892519 24434 tablet_service.cc:1514] Processing DeleteTablet for tablet cb8956fc97b944b9ad1775ac6345d73f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:32 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:50922
I20260619 11:05:32.892902 24693 tablet_replica.cc:330] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: stopping tablet replica
I20260619 11:05:32.892933 24692 tablet_replica.cc:330] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: stopping tablet replica
I20260619 11:05:32.893172 24693 raft_consensus.cc:2229] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:32.893182 24691 tablet_replica.cc:330] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: stopping tablet replica
I20260619 11:05:32.893437 24693 raft_consensus.cc:2258] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:32.893460 24691 raft_consensus.cc:2229] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:32.893671 24155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:05:32.893754 24691 raft_consensus.cc:2258] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:05:32.894078 24155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:32.894546 24692 raft_consensus.cc:2229] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:32.894958 24692 raft_consensus.cc:2258] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:32.894979 24693 ts_tablet_manager.cc:1905] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:32.896238 24692 ts_tablet_manager.cc:1905] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:32.897568 24693 ts_tablet_manager.cc:1918] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:32.897661 24693 log.cc:1196] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/cb8956fc97b944b9ad1775ac6345d73f
I20260619 11:05:32.897964 24693 ts_tablet_manager.cc:1939] T cb8956fc97b944b9ad1775ac6345d73f P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting consensus metadata
I20260619 11:05:32.899065 24692 ts_tablet_manager.cc:1918] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:32.899215 24692 log.cc:1196] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/cb8956fc97b944b9ad1775ac6345d73f
I20260619 11:05:32.899281 24143 catalog_manager.cc:4841] TS a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269): tablet cb8956fc97b944b9ad1775ac6345d73f (table test-table [id=707931d1fc8c464fbcb157e0cc94c895]) successfully deleted
I20260619 11:05:32.899504 24434 tablet_service.cc:1467] Processing CreateTablet for tablet 980a2e798164412da3af6a90f9ca634c (DEFAULT_TABLE table=test-table [id=a40417647bef4f43a4f4b33926f7ede2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:32.899701 24692 ts_tablet_manager.cc:1939] T cb8956fc97b944b9ad1775ac6345d73f P e8627de73e2946fcaf7edef45a0c1820: Deleting consensus metadata
I20260619 11:05:32.899748 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980a2e798164412da3af6a90f9ca634c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:32.899793 24294 tablet_service.cc:1467] Processing CreateTablet for tablet 980a2e798164412da3af6a90f9ca634c (DEFAULT_TABLE table=test-table [id=a40417647bef4f43a4f4b33926f7ede2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:32.900036 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980a2e798164412da3af6a90f9ca634c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:32.901031 24141 catalog_manager.cc:4841] TS e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493): tablet cb8956fc97b944b9ad1775ac6345d73f (table test-table [id=707931d1fc8c464fbcb157e0cc94c895]) successfully deleted
I20260619 11:05:32.901479 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 980a2e798164412da3af6a90f9ca634c (DEFAULT_TABLE table=test-table [id=a40417647bef4f43a4f4b33926f7ede2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:32.901687 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980a2e798164412da3af6a90f9ca634c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:32.902104 24670 tablet_bootstrap.cc:492] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: Bootstrap starting.
I20260619 11:05:32.902967 24670 tablet_bootstrap.cc:654] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:32.903437 24669 tablet_bootstrap.cc:492] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: Bootstrap starting.
I20260619 11:05:32.903867 24668 tablet_bootstrap.cc:492] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: Bootstrap starting.
I20260619 11:05:32.904372 24669 tablet_bootstrap.cc:654] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:32.904726 24668 tablet_bootstrap.cc:654] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:32.905022 24691 ts_tablet_manager.cc:1905] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:32.905866 24669 tablet_bootstrap.cc:492] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: No bootstrap required, opened a new log
I20260619 11:05:32.905943 24669 ts_tablet_manager.cc:1397] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 11:05:32.906045 24668 tablet_bootstrap.cc:492] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: No bootstrap required, opened a new log
I20260619 11:05:32.906113 24668 ts_tablet_manager.cc:1397] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:32.906412 24669 raft_consensus.cc:348] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.906522 24669 raft_consensus.cc:374] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:32.906558 24669 raft_consensus.cc:729] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8627de73e2946fcaf7edef45a0c1820, State: Initialized, Role: FOLLOWER
I20260619 11:05:32.906513 24668 raft_consensus.cc:348] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.906605 24668 raft_consensus.cc:374] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:32.906634 24668 raft_consensus.cc:729] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da958cb952247628042110e3b1104a1, State: Initialized, Role: FOLLOWER
I20260619 11:05:32.906694 24669 consensus_queue.cc:260] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.906733 24668 consensus_queue.cc:260] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.906936 24669 ts_tablet_manager.cc:1428] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:32.906958 24668 ts_tablet_manager.cc:1428] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:32.908746 24691 ts_tablet_manager.cc:1918] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:32.908840 24691 log.cc:1196] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/cb8956fc97b944b9ad1775ac6345d73f
I20260619 11:05:32.909148 24691 ts_tablet_manager.cc:1939] T cb8956fc97b944b9ad1775ac6345d73f P 8da958cb952247628042110e3b1104a1: Deleting consensus metadata
I20260619 11:05:32.910099 24144 catalog_manager.cc:4841] TS 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437): tablet cb8956fc97b944b9ad1775ac6345d73f (table test-table [id=707931d1fc8c464fbcb157e0cc94c895]) successfully deleted
I20260619 11:05:32.910607 24670 tablet_bootstrap.cc:492] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: No bootstrap required, opened a new log
I20260619 11:05:32.910676 24670 ts_tablet_manager.cc:1397] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260619 11:05:32.911124 24670 raft_consensus.cc:348] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.911248 24670 raft_consensus.cc:374] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:32.911288 24670 raft_consensus.cc:729] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fe6508b8c7401a93eeaac98bdf2d1d, State: Initialized, Role: FOLLOWER
I20260619 11:05:32.911417 24670 consensus_queue.cc:260] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.911715 24670 ts_tablet_manager.cc:1428] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:32.959434 24675 raft_consensus.cc:482] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:32.959573 24675 raft_consensus.cc:504] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.960731 24675 leader_election.cc:288] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437), e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493)
I20260619 11:05:32.965660 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.961174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:59497 (local address 127.16.61.1:33437)
0619 11:05:32.961278 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.961282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.961529 (+ 247us) server_negotiation.cc:407] Connection header received
0619 11:05:32.961690 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.961693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.961730 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.961773 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:32.962188 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.962982 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.964497 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.965078 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.965184 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.965244 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.965471 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.965544 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:05:32.965922 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.961438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:56431 (local address 127.16.61.3:40493)
0619 11:05:32.961546 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:32.961551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:32.961779 (+ 228us) server_negotiation.cc:407] Connection header received
0619 11:05:32.961962 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:32.961965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:32.962018 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:32.962063 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:32.962431 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.963148 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.964634 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.965224 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.965315 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.965403 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:32.965718 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 11:05:32.965786 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:05:32.966022 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.960910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33437 (local address 127.16.61.2:59497)
0619 11:05:32.961453 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.961466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.961590 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.961947 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.961950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.961965 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:32.962084 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.962089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.963132 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.963135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.964375 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.964383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.965209 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.965213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.965321 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.965348 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.965809 (+ 461us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.965880 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":62,"threads_started":1}
I20260619 11:05:32.966420 24698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:32.961097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40493 (local address 127.16.61.2:56431)
0619 11:05:32.961703 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:32.961716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:32.961839 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:32.962184 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:32.962187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:32.962206 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:32.962316 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.962321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.963281 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.963284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.964508 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:32.964518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:32.965461 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:32.965467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:32.965573 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:32.965603 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 11:05:32.966196 (+ 593us) client_negotiation.cc:240] Negotiation successful
0619 11:05:32.966263 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260619 11:05:32.966487 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980a2e798164412da3af6a90f9ca634c" candidate_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da958cb952247628042110e3b1104a1" is_pre_election: true
I20260619 11:05:32.966634 24314 raft_consensus.cc:2454] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6fe6508b8c7401a93eeaac98bdf2d1d in term 0.
I20260619 11:05:32.966902 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980a2e798164412da3af6a90f9ca634c" candidate_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8627de73e2946fcaf7edef45a0c1820" is_pre_election: true
I20260619 11:05:32.966960 24388 leader_election.cc:302] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da958cb952247628042110e3b1104a1, a6fe6508b8c7401a93eeaac98bdf2d1d; no voters:
I20260619 11:05:32.967125 24594 raft_consensus.cc:2454] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6fe6508b8c7401a93eeaac98bdf2d1d in term 0.
I20260619 11:05:32.967188 24675 raft_consensus.cc:2781] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:32.967252 24675 raft_consensus.cc:482] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:32.967290 24675 raft_consensus.cc:3037] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:32.968468 24675 raft_consensus.cc:504] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.968797 24675 leader_election.cc:288] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [CANDIDATE]: Term 1 election: Requested vote from peers 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437), e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493)
I20260619 11:05:32.969098 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980a2e798164412da3af6a90f9ca634c" candidate_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da958cb952247628042110e3b1104a1"
I20260619 11:05:32.969170 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980a2e798164412da3af6a90f9ca634c" candidate_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8627de73e2946fcaf7edef45a0c1820"
I20260619 11:05:32.969224 24314 raft_consensus.cc:3037] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:32.969282 24594 raft_consensus.cc:3037] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:32.970052 24314 raft_consensus.cc:2454] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6fe6508b8c7401a93eeaac98bdf2d1d in term 1.
I20260619 11:05:32.970307 24388 leader_election.cc:302] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da958cb952247628042110e3b1104a1, a6fe6508b8c7401a93eeaac98bdf2d1d; no voters:
I20260619 11:05:32.970384 24594 raft_consensus.cc:2454] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6fe6508b8c7401a93eeaac98bdf2d1d in term 1.
I20260619 11:05:32.970562 24675 raft_consensus.cc:2781] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:32.970757 24675 raft_consensus.cc:686] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 LEADER]: Becoming Leader. State: Replica: a6fe6508b8c7401a93eeaac98bdf2d1d, State: Running, Role: LEADER
I20260619 11:05:32.970995 24675 consensus_queue.cc:237] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:32.973237 24156 catalog_manager.cc:5495] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2). New cstate: current_term: 1 leader_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
Jun 19 11:05:33 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867131, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.62@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:33.022127 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.013387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.16.61.62:46037)
0619 11:05:33.013610 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.013625 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.013720 (+ 95us) server_negotiation.cc:407] Connection header received
0619 11:05:33.013852 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.013854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.013891 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.013965 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:33.014762 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.015270 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.016392 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.016608 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.016685 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.018984 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:33.018987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:33.018989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:33.019007 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:33.020514 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:33.020930 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:33.020936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:33.020938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:33.021012 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:33.021289 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:33.021295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:33.021297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:33.021454 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:33.021551 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.021826 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.021976 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260619 11:05:33.022127 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.013315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46037 (local address 127.0.0.1:47498)
0619 11:05:33.013626 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.013642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:33.013758 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:33.014017 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:33.014020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:33.014521 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:33.014643 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.014652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.015378 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.015382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.016270 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.016280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.016720 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.016724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.016811 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.017357 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:33.017372 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:33.018857 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:33.020612 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:33.020622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:33.020628 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:33.020841 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:33.021100 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:33.021103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:33.021105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:33.021208 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:33.021560 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:33.021568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:33.021666 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 11:05:33.021770 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 11:05:33.021922 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":114,"threads_started":1}
I20260619 11:05:33.025701 24156 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:47498:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 11:05:33.026414 24156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:33.030831 24294 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc679617f054430ad11faa63ac756f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93a1130d85794e6ab8c67ab4f353cb91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:33.030930 24434 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc679617f054430ad11faa63ac756f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93a1130d85794e6ab8c67ab4f353cb91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:33.031203 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc679617f054430ad11faa63ac756f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.031203 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc679617f054430ad11faa63ac756f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.033946 24670 tablet_bootstrap.cc:492] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: Bootstrap starting.
I20260619 11:05:33.033969 24668 tablet_bootstrap.cc:492] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: Bootstrap starting.
I20260619 11:05:33.034350 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 1cc679617f054430ad11faa63ac756f7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93a1130d85794e6ab8c67ab4f353cb91]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 11:05:33.034679 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cc679617f054430ad11faa63ac756f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.034909 24668 tablet_bootstrap.cc:654] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.034921 24670 tablet_bootstrap.cc:654] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.036355 24668 tablet_bootstrap.cc:492] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: No bootstrap required, opened a new log
I20260619 11:05:33.036439 24668 ts_tablet_manager.cc:1397] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:05:33.036893 24668 raft_consensus.cc:348] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.037004 24668 raft_consensus.cc:374] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.037034 24668 raft_consensus.cc:729] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da958cb952247628042110e3b1104a1, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.037144 24668 consensus_queue.cc:260] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.037283 24669 tablet_bootstrap.cc:492] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: Bootstrap starting.
I20260619 11:05:33.037374 24668 ts_tablet_manager.cc:1428] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:33.037563 24674 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.037675 24674 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:33.038203 24669 tablet_bootstrap.cc:654] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.038245 24670 tablet_bootstrap.cc:492] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: No bootstrap required, opened a new log
I20260619 11:05:33.038316 24670 ts_tablet_manager.cc:1397] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 11:05:33.038762 24670 raft_consensus.cc:348] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.038865 24670 raft_consensus.cc:374] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.038895 24670 raft_consensus.cc:729] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fe6508b8c7401a93eeaac98bdf2d1d, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.039006 24670 consensus_queue.cc:260] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.039285 24670 ts_tablet_manager.cc:1428] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:33.039933 24669 tablet_bootstrap.cc:492] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: No bootstrap required, opened a new log
I20260619 11:05:33.039875 24675 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.039994 24675 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:33.040012 24669 ts_tablet_manager.cc:1397] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:05:33.040349 24669 raft_consensus.cc:348] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.040424 24669 raft_consensus.cc:374] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.040444 24669 raft_consensus.cc:729] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8627de73e2946fcaf7edef45a0c1820, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.040558 24669 consensus_queue.cc:260] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.040840 24669 ts_tablet_manager.cc:1428] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:05:33.041080 24686 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.041203 24686 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:33.049772 24674 raft_consensus.cc:482] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:33.049858 24674 raft_consensus.cc:504] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.050870 24674 leader_election.cc:288] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269), e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493)
I20260619 11:05:33.054749 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.050997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:37269 (local address 127.16.61.1:42897)
0619 11:05:33.051260 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.051274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:33.051403 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:33.051796 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:33.051800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:33.051836 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:33.051965 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.051970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.052630 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.052635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.053509 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.053517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.054339 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.054342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.054507 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.054526 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:05:33.054589 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 11:05:33.054639 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":23,"spinlock_wait_cycles":34432,"thread_start_us":91,"threads_started":1}
I20260619 11:05:33.054787 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.051095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:42897 (local address 127.16.61.2:37269)
0619 11:05:33.051270 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.051275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.051337 (+ 62us) server_negotiation.cc:407] Connection header received
0619 11:05:33.051518 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.051524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.051573 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.051623 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:33.052061 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.052529 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.053622 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.054216 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.054313 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.054378 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.054619 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.054683 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260619 11:05:33.055203 24454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc679617f054430ad11faa63ac756f7" candidate_uuid: "8da958cb952247628042110e3b1104a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" is_pre_election: true
I20260619 11:05:33.055326 24454 raft_consensus.cc:2454] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da958cb952247628042110e3b1104a1 in term 0.
I20260619 11:05:33.055631 24247 leader_election.cc:302] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da958cb952247628042110e3b1104a1, a6fe6508b8c7401a93eeaac98bdf2d1d; no voters:
I20260619 11:05:33.055871 24674 raft_consensus.cc:2781] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:33.055932 24674 raft_consensus.cc:482] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:33.055997 24674 raft_consensus.cc:3037] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.056058 24665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.051293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:43265 (local address 127.16.61.3:40493)
0619 11:05:33.051387 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.051391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.051807 (+ 416us) server_negotiation.cc:407] Connection header received
0619 11:05:33.051928 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.051930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.051968 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.052033 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:33.052895 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.053561 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.055074 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.055536 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.055643 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.055713 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.055877 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.055940 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 11:05:33.056135 24712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.051177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40493 (local address 127.16.61.1:43265)
0619 11:05:33.051710 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.051724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:33.051836 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:33.052659 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:33.052664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:33.052682 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:33.052782 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.052787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.053677 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.053680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.054978 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.054990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.055644 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.055648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.055762 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.055786 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:05:33.055865 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:05:33.055926 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":260,"spinlock_wait_cycles":2432,"thread_start_us":89,"threads_started":1}
I20260619 11:05:33.057206 24674 raft_consensus.cc:504] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.057539 24674 leader_election.cc:288] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [CANDIDATE]: Term 1 election: Requested vote from peers a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269), e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493)
I20260619 11:05:33.057794 24454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc679617f054430ad11faa63ac756f7" candidate_uuid: "8da958cb952247628042110e3b1104a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d"
I20260619 11:05:33.057898 24454 raft_consensus.cc:3037] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.057895 24594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc679617f054430ad11faa63ac756f7" candidate_uuid: "8da958cb952247628042110e3b1104a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8627de73e2946fcaf7edef45a0c1820" is_pre_election: true
I20260619 11:05:33.057932 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cc679617f054430ad11faa63ac756f7" candidate_uuid: "8da958cb952247628042110e3b1104a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8627de73e2946fcaf7edef45a0c1820"
I20260619 11:05:33.058027 24594 raft_consensus.cc:2454] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da958cb952247628042110e3b1104a1 in term 0.
I20260619 11:05:33.058911 24454 raft_consensus.cc:2454] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da958cb952247628042110e3b1104a1 in term 1.
I20260619 11:05:33.059154 24247 leader_election.cc:302] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8da958cb952247628042110e3b1104a1, a6fe6508b8c7401a93eeaac98bdf2d1d; no voters: e8627de73e2946fcaf7edef45a0c1820
I20260619 11:05:33.059332 24674 raft_consensus.cc:2781] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:33.059538 24674 raft_consensus.cc:686] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8da958cb952247628042110e3b1104a1, State: Running, Role: LEADER
I20260619 11:05:33.059778 24674 consensus_queue.cc:237] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.060567 24713 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: TxnStatusTablet state changed. Reason: New leader 8da958cb952247628042110e3b1104a1. Latest consensus state: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.060672 24713 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:33.061096 24715 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 11:05:33.061236 24715 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 11:05:33.061618 24155 catalog_manager.cc:5495] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8da958cb952247628042110e3b1104a1 (127.16.61.1). New cstate: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:33.071605 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.068402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.16.61.1:33437)
0619 11:05:33.068495 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.068498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.068510 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:33.068684 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.068687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.068737 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.068829 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:33.069052 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.069530 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.070341 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.070553 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.070650 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.070925 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.071262 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.071389 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.071454 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.071497 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 11:05:33.071638 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.068341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33437 (local address 127.0.0.1:51760)
0619 11:05:33.068454 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.068470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:33.068559 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:33.068846 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:33.068849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:33.068858 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:33.068952 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.068962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.069630 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.069634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.070224 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.070231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.070660 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.070664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.070747 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.070792 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.071365 (+ 573us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.071379 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 11:05:33.071446 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 11:05:33.071523 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 11:05:33.074147 24454 raft_consensus.cc:1261] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Refusing update from remote peer 8da958cb952247628042110e3b1104a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:33.074169 24593 raft_consensus.cc:3037] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.074535 24674 consensus_queue.cc:1035] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:33.075404 24593 raft_consensus.cc:1261] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Refusing update from remote peer 8da958cb952247628042110e3b1104a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:33.075918 24675 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: TxnStatusTablet state changed. Reason: New leader 8da958cb952247628042110e3b1104a1. Latest consensus state: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.076021 24675 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:33.075981 24674 consensus_queue.cc:1035] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:33.077692 24674 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.077792 24674 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:33.077952 24686 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: TxnStatusTablet state changed. Reason: New leader 8da958cb952247628042110e3b1104a1. Latest consensus state: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.078037 24686 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:33.079748 24675 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.079842 24675 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P a6fe6508b8c7401a93eeaac98bdf2d1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:33.079999 24683 mvcc.cc:205] Tried to move back new op lower bound from 7298527777067573248 to 7298527777014837248. Current Snapshot: MvccSnapshot[applied={T|T < 7298527777067573248 or (T in {7298527777067573248})}]
I20260619 11:05:33.081156 24713 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.081270 24713 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P 8da958cb952247628042110e3b1104a1: This TxnStatusTablet replica's current role is: LEADER
I20260619 11:05:33.082367 24681 mvcc.cc:205] Tried to move back new op lower bound from 7298527777067573248 to 7298527777014837248. Current Snapshot: MvccSnapshot[applied={T|T < 7298527777067573248}]
I20260619 11:05:33.082911 24686 tablet_replica.cc:439] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } } }
I20260619 11:05:33.082985 24686 tablet_replica.cc:442] T 1cc679617f054430ad11faa63ac756f7 P e8627de73e2946fcaf7edef45a0c1820: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 11:05:33.087844 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.084205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.16.61.2:37269)
0619 11:05:33.084322 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.084326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.084567 (+ 241us) server_negotiation.cc:407] Connection header received
0619 11:05:33.084621 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.084623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.084665 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.084733 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:33.084983 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.085450 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.086559 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.086767 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.086841 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.087085 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.087437 (+ 352us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.087513 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.087611 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.087648 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:05:33.097247 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.094010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:44225 (local address 127.16.61.1:33437)
0619 11:05:33.094115 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.094119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.094307 (+ 188us) server_negotiation.cc:407] Connection header received
0619 11:05:33.094422 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.094424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.094458 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.094516 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:33.094753 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.095194 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.096379 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.096571 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.096646 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.096915 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.096967 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.097017 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.097123 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.097157 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 11:05:33.097321 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.093948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33437 (local address 127.16.61.2:44225)
0619 11:05:33.094227 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.094241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:33.094335 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:33.094542 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:33.094545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:33.094553 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:33.094654 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.094659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.095283 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.095287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.096275 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.096285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.096667 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.096671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.096763 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.096806 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.097041 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.097047 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:05:33.097111 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 11:05:33.097177 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
I20260619 11:05:33.103765 24593 raft_consensus.cc:1261] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Refusing update from remote peer a6fe6508b8c7401a93eeaac98bdf2d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:33.103868 24314 raft_consensus.cc:1261] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Refusing update from remote peer a6fe6508b8c7401a93eeaac98bdf2d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:33.104198 24675 consensus_queue.cc:1035] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:33.104617 24699 consensus_queue.cc:1035] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:33.124732 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.120958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:42847 (local address 127.16.61.2:37269)
0619 11:05:33.121051 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.121057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.121283 (+ 226us) server_negotiation.cc:407] Connection header received
0619 11:05:33.121419 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.121422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.121466 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.121540 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:33.121801 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.122488 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.123791 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.123995 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.124071 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.124351 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.124407 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.124460 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.124560 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.124597 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 11:05:33.124779 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.120874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:37269 (local address 127.16.61.1:42847)
0619 11:05:33.121206 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.121217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:33.121324 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:33.121549 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:33.121552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:33.121559 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:33.121683 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.121688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.122579 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.122582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.123689 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.123700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.124091 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.124095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.124188 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.124234 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.124480 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.124486 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:05:33.124555 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:05:33.124624 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260619 11:05:33.146947 24156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:33.147100 24156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:33.149298 24156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: Sending DeleteTablet for 3 replicas of tablet 980a2e798164412da3af6a90f9ca634c
I20260619 11:05:33.149776 24294 tablet_service.cc:1514] Processing DeleteTablet for tablet 980a2e798164412da3af6a90f9ca634c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:51756
I20260619 11:05:33.149912 24434 tablet_service.cc:1514] Processing DeleteTablet for tablet 980a2e798164412da3af6a90f9ca634c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:50922
I20260619 11:05:33.150022 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 980a2e798164412da3af6a90f9ca634c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:52254
I20260619 11:05:33.150068 24691 tablet_replica.cc:330] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: stopping tablet replica
I20260619 11:05:33.150177 24691 raft_consensus.cc:2229] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:33.150302 24691 raft_consensus.cc:2258] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.150472 24693 tablet_replica.cc:330] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: stopping tablet replica
I20260619 11:05:33.150588 24693 raft_consensus.cc:2229] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:33.150820 24693 raft_consensus.cc:2258] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.150997 24692 tablet_replica.cc:330] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: stopping tablet replica
I20260619 11:05:33.151119 24692 raft_consensus.cc:2229] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:33.151232 24692 raft_consensus.cc:2258] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.151496 24693 ts_tablet_manager.cc:1905] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:33.151891 24692 ts_tablet_manager.cc:1905] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:33.151981 24691 ts_tablet_manager.cc:1905] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:33.154480 24691 ts_tablet_manager.cc:1918] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:33.154551 24691 log.cc:1196] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/980a2e798164412da3af6a90f9ca634c
I20260619 11:05:33.154805 24691 ts_tablet_manager.cc:1939] T 980a2e798164412da3af6a90f9ca634c P 8da958cb952247628042110e3b1104a1: Deleting consensus metadata
I20260619 11:05:33.157748 24144 catalog_manager.cc:4841] TS 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437): tablet 980a2e798164412da3af6a90f9ca634c (table test-table [id=a40417647bef4f43a4f4b33926f7ede2]) successfully deleted
I20260619 11:05:33.158382 24693 ts_tablet_manager.cc:1918] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:33.158466 24693 log.cc:1196] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/980a2e798164412da3af6a90f9ca634c
I20260619 11:05:33.158865 24693 ts_tablet_manager.cc:1939] T 980a2e798164412da3af6a90f9ca634c P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting consensus metadata
I20260619 11:05:33.159514 24143 catalog_manager.cc:4841] TS a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269): tablet 980a2e798164412da3af6a90f9ca634c (table test-table [id=a40417647bef4f43a4f4b33926f7ede2]) successfully deleted
I20260619 11:05:33.160059 24692 ts_tablet_manager.cc:1918] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:33.160130 24692 log.cc:1196] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/980a2e798164412da3af6a90f9ca634c
I20260619 11:05:33.160367 24692 ts_tablet_manager.cc:1939] T 980a2e798164412da3af6a90f9ca634c P e8627de73e2946fcaf7edef45a0c1820: Deleting consensus metadata
I20260619 11:05:33.161015 24141 catalog_manager.cc:4841] TS e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493): tablet 980a2e798164412da3af6a90f9ca634c (table test-table [id=a40417647bef4f43a4f4b33926f7ede2]) successfully deleted
Jun 19 11:05:33 dist-test-slave-8vcz krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 11:05:33 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:33 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:05:33.195629 24707 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:33.188091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.16.61.62:46037)
0619 11:05:33.188206 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.188211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.188480 (+ 269us) server_negotiation.cc:407] Connection header received
0619 11:05:33.188589 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.188592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.188647 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.188735 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:33.189547 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.190370 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.191586 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.191955 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.192081 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.195510 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47504: BlockingRecv error: recv got EOF from 127.0.0.1:47504 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 11:05:33 dist-test-slave-8vcz krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM
I20260619 11:05:33.206902 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.197026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.16.61.62:46037)
0619 11:05:33.197145 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.197149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.197570 (+ 421us) server_negotiation.cc:407] Connection header received
0619 11:05:33.197686 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.197689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.197735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.197811 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:33.198457 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.199158 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.200273 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.200595 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.200697 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.203419 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:33.203424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:33.203427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:33.203452 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:33.205207 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:33.205667 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:33.205671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:33.205673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:33.205760 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:33.206052 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:33.206055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:33.206057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:33.206266 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:33.206413 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.206639 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.206751 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:05:33.209692 24156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:33.210111 24156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:33.214694 24294 tablet_service.cc:1467] Processing CreateTablet for tablet d19f051dfd444bfc988715fefbe991a5 (DEFAULT_TABLE table=test-table [id=7192c8f424de4b128402750e16e3294d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:33.214979 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19f051dfd444bfc988715fefbe991a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.215057 24434 tablet_service.cc:1467] Processing CreateTablet for tablet d19f051dfd444bfc988715fefbe991a5 (DEFAULT_TABLE table=test-table [id=7192c8f424de4b128402750e16e3294d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:33.215297 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19f051dfd444bfc988715fefbe991a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.217446 24668 tablet_bootstrap.cc:492] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: Bootstrap starting.
I20260619 11:05:33.217660 24574 tablet_service.cc:1467] Processing CreateTablet for tablet d19f051dfd444bfc988715fefbe991a5 (DEFAULT_TABLE table=test-table [id=7192c8f424de4b128402750e16e3294d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:33.217912 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d19f051dfd444bfc988715fefbe991a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.218353 24668 tablet_bootstrap.cc:654] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.219848 24670 tablet_bootstrap.cc:492] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: Bootstrap starting.
I20260619 11:05:33.220134 24669 tablet_bootstrap.cc:492] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: Bootstrap starting.
I20260619 11:05:33.220748 24670 tablet_bootstrap.cc:654] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.220882 24669 tablet_bootstrap.cc:654] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.222153 24670 tablet_bootstrap.cc:492] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: No bootstrap required, opened a new log
I20260619 11:05:33.222231 24670 ts_tablet_manager.cc:1397] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:33.222262 24668 tablet_bootstrap.cc:492] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: No bootstrap required, opened a new log
I20260619 11:05:33.222327 24668 ts_tablet_manager.cc:1397] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 11:05:33.222620 24670 raft_consensus.cc:348] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.222731 24670 raft_consensus.cc:374] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.222760 24670 raft_consensus.cc:729] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fe6508b8c7401a93eeaac98bdf2d1d, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.222735 24668 raft_consensus.cc:348] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.222828 24668 raft_consensus.cc:374] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.222855 24668 raft_consensus.cc:729] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da958cb952247628042110e3b1104a1, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.222878 24670 consensus_queue.cc:260] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.222957 24668 consensus_queue.cc:260] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.223100 24670 ts_tablet_manager.cc:1428] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:33.223187 24668 ts_tablet_manager.cc:1428] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:33.228305 24669 tablet_bootstrap.cc:492] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: No bootstrap required, opened a new log
I20260619 11:05:33.228391 24669 ts_tablet_manager.cc:1397] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260619 11:05:33.228741 24669 raft_consensus.cc:348] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.228860 24669 raft_consensus.cc:374] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.228897 24669 raft_consensus.cc:729] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8627de73e2946fcaf7edef45a0c1820, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.229018 24669 consensus_queue.cc:260] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.229272 24669 ts_tablet_manager.cc:1428] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:05:33.386114 24720 raft_consensus.cc:482] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:33.386255 24720 raft_consensus.cc:504] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.386698 24720 leader_election.cc:288] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269), e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493)
I20260619 11:05:33.387135 24454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19f051dfd444bfc988715fefbe991a5" candidate_uuid: "8da958cb952247628042110e3b1104a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" is_pre_election: true
I20260619 11:05:33.387193 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19f051dfd444bfc988715fefbe991a5" candidate_uuid: "8da958cb952247628042110e3b1104a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8627de73e2946fcaf7edef45a0c1820" is_pre_election: true
I20260619 11:05:33.387296 24454 raft_consensus.cc:2454] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da958cb952247628042110e3b1104a1 in term 0.
I20260619 11:05:33.387323 24593 raft_consensus.cc:2454] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8da958cb952247628042110e3b1104a1 in term 0.
I20260619 11:05:33.387626 24245 leader_election.cc:302] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8da958cb952247628042110e3b1104a1, a6fe6508b8c7401a93eeaac98bdf2d1d, e8627de73e2946fcaf7edef45a0c1820; no voters:
I20260619 11:05:33.387804 24720 raft_consensus.cc:2781] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:33.387862 24720 raft_consensus.cc:482] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:33.387912 24720 raft_consensus.cc:3037] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.388832 24720 raft_consensus.cc:504] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.389118 24720 leader_election.cc:288] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [CANDIDATE]: Term 1 election: Requested vote from peers a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269), e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493)
I20260619 11:05:33.389398 24454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19f051dfd444bfc988715fefbe991a5" candidate_uuid: "8da958cb952247628042110e3b1104a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d"
I20260619 11:05:33.389494 24454 raft_consensus.cc:3037] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.389494 24593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d19f051dfd444bfc988715fefbe991a5" candidate_uuid: "8da958cb952247628042110e3b1104a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8627de73e2946fcaf7edef45a0c1820"
I20260619 11:05:33.389626 24593 raft_consensus.cc:3037] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.390379 24454 raft_consensus.cc:2454] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da958cb952247628042110e3b1104a1 in term 1.
I20260619 11:05:33.390542 24593 raft_consensus.cc:2454] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8da958cb952247628042110e3b1104a1 in term 1.
I20260619 11:05:33.390661 24247 leader_election.cc:302] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da958cb952247628042110e3b1104a1, a6fe6508b8c7401a93eeaac98bdf2d1d; no voters:
I20260619 11:05:33.390830 24720 raft_consensus.cc:2781] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:33.390930 24720 raft_consensus.cc:686] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8da958cb952247628042110e3b1104a1, State: Running, Role: LEADER
I20260619 11:05:33.391060 24720 consensus_queue.cc:237] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.392606 24156 catalog_manager.cc:5495] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8da958cb952247628042110e3b1104a1 (127.16.61.1). New cstate: current_term: 1 leader_uuid: "8da958cb952247628042110e3b1104a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:33.415417 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.412671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.16.61.1:33437)
0619 11:05:33.412781 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.412786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.412804 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:05:33.412891 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.412893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.412931 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.413055 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:33.413307 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.413798 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.414599 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.414797 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.414870 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.415094 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.415151 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.415200 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.415277 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.415312 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:05:33.417618 24593 raft_consensus.cc:1261] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Refusing update from remote peer 8da958cb952247628042110e3b1104a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:33.417958 24454 raft_consensus.cc:1261] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Refusing update from remote peer 8da958cb952247628042110e3b1104a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:33.418043 24720 consensus_queue.cc:1035] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:33.418488 24674 consensus_queue.cc:1035] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:33.426883 24156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:33.427013 24156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:33.429111 24156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: Sending DeleteTablet for 3 replicas of tablet d19f051dfd444bfc988715fefbe991a5
I20260619 11:05:33.429594 24294 tablet_service.cc:1514] Processing DeleteTablet for tablet d19f051dfd444bfc988715fefbe991a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:51756
I20260619 11:05:33.429677 24434 tablet_service.cc:1514] Processing DeleteTablet for tablet d19f051dfd444bfc988715fefbe991a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:50922
I20260619 11:05:33.429692 24293 tablet_service.cc:1514] Processing DeleteTablet for tablet d19f051dfd444bfc988715fefbe991a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:51756
I20260619 11:05:33.429752 24691 tablet_replica.cc:330] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: stopping tablet replica
I20260619 11:05:33.429808 24693 tablet_replica.cc:330] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: stopping tablet replica
I20260619 11:05:33.429860 24691 raft_consensus.cc:2229] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:33.429894 24693 raft_consensus.cc:2229] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:33.429967 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet d19f051dfd444bfc988715fefbe991a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:52254
I20260619 11:05:33.430001 24693 raft_consensus.cc:2258] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.430090 24691 raft_consensus.cc:2258] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.430253 24692 tablet_replica.cc:330] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: stopping tablet replica
I20260619 11:05:33.430358 24692 raft_consensus.cc:2229] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:33.430471 24692 raft_consensus.cc:2258] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.430675 24693 ts_tablet_manager.cc:1905] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:33.430799 24156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 11:05:33.431036 24692 ts_tablet_manager.cc:1905] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:05:33.431169 24156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:33.431241 24691 ts_tablet_manager.cc:1905] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:33.433085 24693 ts_tablet_manager.cc:1918] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:33.433149 24693 log.cc:1196] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/d19f051dfd444bfc988715fefbe991a5
I20260619 11:05:33.433418 24693 ts_tablet_manager.cc:1939] T d19f051dfd444bfc988715fefbe991a5 P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting consensus metadata
I20260619 11:05:33.433401 24692 ts_tablet_manager.cc:1918] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:33.433465 24692 log.cc:1196] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/d19f051dfd444bfc988715fefbe991a5
I20260619 11:05:33.433591 24691 ts_tablet_manager.cc:1918] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:33.433655 24691 log.cc:1196] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/d19f051dfd444bfc988715fefbe991a5
I20260619 11:05:33.433709 24692 ts_tablet_manager.cc:1939] T d19f051dfd444bfc988715fefbe991a5 P e8627de73e2946fcaf7edef45a0c1820: Deleting consensus metadata
I20260619 11:05:33.433977 24691 ts_tablet_manager.cc:1939] T d19f051dfd444bfc988715fefbe991a5 P 8da958cb952247628042110e3b1104a1: Deleting consensus metadata
I20260619 11:05:33.434440 24141 catalog_manager.cc:4841] TS e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493): tablet d19f051dfd444bfc988715fefbe991a5 (table test-table [id=7192c8f424de4b128402750e16e3294d]) successfully deleted
I20260619 11:05:33.434593 24144 catalog_manager.cc:4841] TS 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437): tablet d19f051dfd444bfc988715fefbe991a5 (table test-table [id=7192c8f424de4b128402750e16e3294d]) successfully deleted
I20260619 11:05:33.434641 24143 catalog_manager.cc:4841] TS a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269): tablet d19f051dfd444bfc988715fefbe991a5 (table test-table [id=7192c8f424de4b128402750e16e3294d]) successfully deleted
W20260619 11:05:33.434842 24144 catalog_manager.cc:4808] TS 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437): delete failed for tablet d19f051dfd444bfc988715fefbe991a5 because the tablet was not found. No further retry: Not found: Tablet not found: d19f051dfd444bfc988715fefbe991a5
I20260619 11:05:33.435551 24434 tablet_service.cc:1467] Processing CreateTablet for tablet 54745489acdd48cbaf8f13a04d303a08 (DEFAULT_TABLE table=test-table [id=72d73dad76054244bd05c6a227fb8489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:33.435606 24574 tablet_service.cc:1467] Processing CreateTablet for tablet 54745489acdd48cbaf8f13a04d303a08 (DEFAULT_TABLE table=test-table [id=72d73dad76054244bd05c6a227fb8489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:33.435827 24574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54745489acdd48cbaf8f13a04d303a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.435899 24293 tablet_service.cc:1467] Processing CreateTablet for tablet 54745489acdd48cbaf8f13a04d303a08 (DEFAULT_TABLE table=test-table [id=72d73dad76054244bd05c6a227fb8489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:33.436105 24293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54745489acdd48cbaf8f13a04d303a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.437862 24669 tablet_bootstrap.cc:492] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: Bootstrap starting.
I20260619 11:05:33.438192 24668 tablet_bootstrap.cc:492] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: Bootstrap starting.
I20260619 11:05:33.438489 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54745489acdd48cbaf8f13a04d303a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:33.438625 24669 tablet_bootstrap.cc:654] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.439013 24668 tablet_bootstrap.cc:654] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.440018 24668 tablet_bootstrap.cc:492] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: No bootstrap required, opened a new log
I20260619 11:05:33.440071 24669 tablet_bootstrap.cc:492] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: No bootstrap required, opened a new log
I20260619 11:05:33.440084 24668 ts_tablet_manager.cc:1397] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:33.440138 24669 ts_tablet_manager.cc:1397] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:33.440408 24668 raft_consensus.cc:348] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.440500 24668 raft_consensus.cc:374] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.440529 24668 raft_consensus.cc:729] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da958cb952247628042110e3b1104a1, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.440538 24669 raft_consensus.cc:348] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.440640 24669 raft_consensus.cc:374] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.440627 24668 consensus_queue.cc:260] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.440675 24669 raft_consensus.cc:729] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8627de73e2946fcaf7edef45a0c1820, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.440694 24670 tablet_bootstrap.cc:492] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d: Bootstrap starting.
I20260619 11:05:33.440779 24669 consensus_queue.cc:260] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.440845 24668 ts_tablet_manager.cc:1428] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:33.441006 24669 ts_tablet_manager.cc:1428] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 11:05:33.441464 24670 tablet_bootstrap.cc:654] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:33.443197 24670 tablet_bootstrap.cc:492] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d: No bootstrap required, opened a new log
I20260619 11:05:33.443288 24670 ts_tablet_manager.cc:1397] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 11:05:33.443604 24670 raft_consensus.cc:348] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.443763 24670 raft_consensus.cc:374] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:33.443797 24670 raft_consensus.cc:729] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6fe6508b8c7401a93eeaac98bdf2d1d, State: Initialized, Role: FOLLOWER
I20260619 11:05:33.443918 24670 consensus_queue.cc:260] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.444111 24670 ts_tablet_manager.cc:1428] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 11:05:33.587357 24686 raft_consensus.cc:482] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:33.587476 24686 raft_consensus.cc:504] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.587920 24686 leader_election.cc:288] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437), a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269)
I20260619 11:05:33.588230 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54745489acdd48cbaf8f13a04d303a08" candidate_uuid: "e8627de73e2946fcaf7edef45a0c1820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da958cb952247628042110e3b1104a1" is_pre_election: true
I20260619 11:05:33.588378 24314 raft_consensus.cc:2454] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8627de73e2946fcaf7edef45a0c1820 in term 0.
I20260619 11:05:33.588318 24454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54745489acdd48cbaf8f13a04d303a08" candidate_uuid: "e8627de73e2946fcaf7edef45a0c1820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" is_pre_election: true
I20260619 11:05:33.588510 24454 raft_consensus.cc:2454] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8627de73e2946fcaf7edef45a0c1820 in term 0.
I20260619 11:05:33.588670 24528 leader_election.cc:302] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da958cb952247628042110e3b1104a1, e8627de73e2946fcaf7edef45a0c1820; no voters:
I20260619 11:05:33.588847 24686 raft_consensus.cc:2781] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:33.588909 24686 raft_consensus.cc:482] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:33.588960 24686 raft_consensus.cc:3037] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.589921 24686 raft_consensus.cc:504] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.590265 24686 leader_election.cc:288] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [CANDIDATE]: Term 1 election: Requested vote from peers 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437), a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269)
I20260619 11:05:33.590503 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54745489acdd48cbaf8f13a04d303a08" candidate_uuid: "e8627de73e2946fcaf7edef45a0c1820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da958cb952247628042110e3b1104a1"
I20260619 11:05:33.590629 24314 raft_consensus.cc:3037] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.590600 24454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54745489acdd48cbaf8f13a04d303a08" candidate_uuid: "e8627de73e2946fcaf7edef45a0c1820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d"
I20260619 11:05:33.590728 24454 raft_consensus.cc:3037] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:33.591507 24314 raft_consensus.cc:2454] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8627de73e2946fcaf7edef45a0c1820 in term 1.
I20260619 11:05:33.591571 24454 raft_consensus.cc:2454] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8627de73e2946fcaf7edef45a0c1820 in term 1.
I20260619 11:05:33.591794 24528 leader_election.cc:302] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da958cb952247628042110e3b1104a1, e8627de73e2946fcaf7edef45a0c1820; no voters:
I20260619 11:05:33.591957 24686 raft_consensus.cc:2781] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:33.592147 24686 raft_consensus.cc:686] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 1 LEADER]: Becoming Leader. State: Replica: e8627de73e2946fcaf7edef45a0c1820, State: Running, Role: LEADER
I20260619 11:05:33.592280 24686 consensus_queue.cc:237] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } }
I20260619 11:05:33.593708 24156 catalog_manager.cc:5495] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3). New cstate: current_term: 1 leader_uuid: "e8627de73e2946fcaf7edef45a0c1820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8627de73e2946fcaf7edef45a0c1820" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40493 } health_report { overall_health: HEALTHY } } }
I20260619 11:05:33.640008 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.636904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.16.61.3:40493)
0619 11:05:33.637126 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.637149 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.637162 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:33.637200 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.637203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.637249 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.637333 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:33.637577 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.638082 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.638960 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.639197 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.639301 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.639538 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.639613 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.639711 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.639802 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.639846 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260619 11:05:33.648280 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.645593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:46137 (local address 127.16.61.1:33437)
0619 11:05:33.645707 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.645711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.645884 (+ 173us) server_negotiation.cc:407] Connection header received
0619 11:05:33.645962 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.645963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.646002 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.646059 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:33.646211 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.646655 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.647413 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.647606 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.647732 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.647971 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.648027 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.648081 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.648151 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.648186 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:05:33.648299 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.645536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33437 (local address 127.16.61.3:46137)
0619 11:05:33.645801 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.645813 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:33.645892 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:33.646074 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:33.646076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:33.646081 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:33.646152 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.646155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.646728 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.646730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.647321 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.647328 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.647736 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.647739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.647802 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.647838 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.648093 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.648098 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:05:33.648144 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 11:05:33.648198 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260619 11:05:33.652868 24454 raft_consensus.cc:1261] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Refusing update from remote peer e8627de73e2946fcaf7edef45a0c1820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:33.652920 24314 raft_consensus.cc:1261] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Refusing update from remote peer e8627de73e2946fcaf7edef45a0c1820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:33.653225 24756 consensus_queue.cc:1035] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6fe6508b8c7401a93eeaac98bdf2d1d" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:33.653405 24686 consensus_queue.cc:1035] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da958cb952247628042110e3b1104a1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:33.668726 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.665533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:56653 (local address 127.16.61.3:40493)
0619 11:05:33.665626 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:33.665630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:33.665843 (+ 213us) server_negotiation.cc:407] Connection header received
0619 11:05:33.665945 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:33.665947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:33.665989 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:33.666053 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:33.666316 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.666832 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.667596 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.667929 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.668037 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.668292 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.668350 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.668433 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:33.668546 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 11:05:33.668596 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 11:05:33.668744 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.665461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40493 (local address 127.16.61.1:56653)
0619 11:05:33.665758 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.665772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:33.665865 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:33.666141 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:33.666144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:33.666152 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 11:05:33.666236 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.666241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.666930 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.666933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.667509 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:33.667516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:33.668050 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:33.668054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:33.668146 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:33.668180 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 11:05:33.668443 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 11:05:33.668450 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:05:33.668521 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:05:33.668599 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":69,"threads_started":1}
I20260619 11:05:33.687903 24156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:33.688057 24156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:33.689735 24156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08120b984a6a48ae94ad3aa04690799b: Sending DeleteTablet for 3 replicas of tablet 54745489acdd48cbaf8f13a04d303a08
I20260619 11:05:33.690259 24293 tablet_service.cc:1514] Processing DeleteTablet for tablet 54745489acdd48cbaf8f13a04d303a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:51756
I20260619 11:05:33.690352 24574 tablet_service.cc:1514] Processing DeleteTablet for tablet 54745489acdd48cbaf8f13a04d303a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:52254
I20260619 11:05:33.690521 24691 tablet_replica.cc:330] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: stopping tablet replica
I20260619 11:05:33.690641 24691 raft_consensus.cc:2229] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:33.690760 24691 raft_consensus.cc:2258] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.691313 24691 ts_tablet_manager.cc:1905] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:33.693378 24691 ts_tablet_manager.cc:1918] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:33.693470 24691 log.cc:1196] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/54745489acdd48cbaf8f13a04d303a08
I20260619 11:05:33.693694 24691 ts_tablet_manager.cc:1939] T 54745489acdd48cbaf8f13a04d303a08 P 8da958cb952247628042110e3b1104a1: Deleting consensus metadata
I20260619 11:05:33.694249 24144 catalog_manager.cc:4841] TS 8da958cb952247628042110e3b1104a1 (127.16.61.1:33437): tablet 54745489acdd48cbaf8f13a04d303a08 (table test-table [id=72d73dad76054244bd05c6a227fb8489]) successfully deleted
I20260619 11:05:33.695761 24434 tablet_service.cc:1514] Processing DeleteTablet for tablet 54745489acdd48cbaf8f13a04d303a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:33 UTC) from {username='oryx', principal='oryx/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:50922
I20260619 11:05:33.695813 24692 tablet_replica.cc:330] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: stopping tablet replica
I20260619 11:05:33.695930 24692 raft_consensus.cc:2229] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:33.695940 24693 tablet_replica.cc:330] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d: stopping tablet replica
I20260619 11:05:33.696058 24693 raft_consensus.cc:2229] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:33.696137 24692 raft_consensus.cc:2258] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.696166 24693 raft_consensus.cc:2258] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:33.696839 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 24229
I20260619 11:05:33.697016 24692 ts_tablet_manager.cc:1905] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:33.698895 24692 ts_tablet_manager.cc:1918] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 11:05:33.698954 24692 log.cc:1196] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/54745489acdd48cbaf8f13a04d303a08
I20260619 11:05:33.699129 24692 ts_tablet_manager.cc:1939] T 54745489acdd48cbaf8f13a04d303a08 P e8627de73e2946fcaf7edef45a0c1820: Deleting consensus metadata
I20260619 11:05:33.700186 24141 catalog_manager.cc:4841] TS e8627de73e2946fcaf7edef45a0c1820 (127.16.61.3:40493): tablet 54745489acdd48cbaf8f13a04d303a08 (table test-table [id=72d73dad76054244bd05c6a227fb8489]) successfully deleted
I20260619 11:05:33.704290 24693 ts_tablet_manager.cc:1905] T 54745489acdd48cbaf8f13a04d303a08 P a6fe6508b8c7401a93eeaac98bdf2d1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:33.706476 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 24369
W20260619 11:05:33.712706 24143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.2:37269 (error 108)
I20260619 11:05:33.712942 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 24509
I20260619 11:05:33.714006 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:33.713647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:37269 (local address 127.0.0.1:50934)
0619 11:05:33.713843 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:33.713931 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.2:37269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
W20260619 11:05:33.714141 24143 catalog_manager.cc:4584] TS a6fe6508b8c7401a93eeaac98bdf2d1d (127.16.61.2:37269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54745489acdd48cbaf8f13a04d303a08: Network error: Client connection negotiation failed: client connection to 127.16.61.2:37269: connect: Connection refused (error 111)
I20260619 11:05:33.720083 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 24125
2026-06-19T11:05:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3375 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:33 dist-test-slave-8vcz krb5kdc[24772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:33 dist-test-slave-8vcz krb5kdc[24772](info): set up 2 sockets
Jun 19 11:05:33 dist-test-slave-8vcz krb5kdc[24772](info): commencing operation
krb5kdc: starting...
W20260619 11:05:34.756348 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:34 dist-test-slave-8vcz krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:34.870323 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44747
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:44747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:34.972821 24785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:34.973045 24785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:34.973084 24785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:34.977001 24785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:34.977047 24785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:34.977065 24785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:34.977082 24785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:34.977097 24785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:34.981942 24785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:44747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:34.982754 24785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:34.983585 24785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:34.991335 24790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:34.991401 24785 server_base.cc:1034] running on GCE node
W20260619 11:05:34.991466 24791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:34.991354 24793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:34.992074 24785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:34.993063 24785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:34.994259 24785 hybrid_clock.cc:648] HybridClock initialized: now 1781867134994237 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T11:05:35Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1276 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:35 dist-test-slave-8vcz krb5kdc[24798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:35 dist-test-slave-8vcz krb5kdc[24798](info): set up 2 sockets
Jun 19 11:05:35 dist-test-slave-8vcz krb5kdc[24798](info): commencing operation
krb5kdc: starting...
W20260619 11:05:36.024299 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:36.134572 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44377
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:36.237648 24811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:36.237917 24811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:36.237989 24811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:36.241849 24811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:36.241930 24811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:36.241973 24811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:36.242024 24811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:36.242063 24811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:36.247057 24811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:36.248025 24811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:36.248936 24811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:36.256672 24819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:36.256713 24816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:36.256687 24817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:36.256862 24811 server_base.cc:1034] running on GCE node
I20260619 11:05:36.257440 24811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:36.258455 24811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:36.259651 24811 hybrid_clock.cc:648] HybridClock initialized: now 1781867136259622 us; error 41 us; skew 500 ppm
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:36.262272 24811 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:36.263079 24811 webserver.cc:460] Webserver started at http://127.16.61.62:43341/ using document root <none> and password file <none>
I20260619 11:05:36.263610 24811 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:36.263717 24811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:36.263942 24811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:36.265944 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "8c1ac342eb0141ffbf37ecb90578b24f"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "aa3507e23fd0b93472f0d9d0146dcf2b"
server_key_iv: "2a80e452597b87cb3fb543e6d9115212"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.266469 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "8c1ac342eb0141ffbf37ecb90578b24f"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "aa3507e23fd0b93472f0d9d0146dcf2b"
server_key_iv: "2a80e452597b87cb3fb543e6d9115212"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.270006 24811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:05:36.272253 24826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.273365 24811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:05:36.273490 24811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "8c1ac342eb0141ffbf37ecb90578b24f"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "aa3507e23fd0b93472f0d9d0146dcf2b"
server_key_iv: "2a80e452597b87cb3fb543e6d9115212"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.273590 24811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:36.288244 24811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:36.291074 24811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:36.291260 24811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:36.297714 24811 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:44377
I20260619 11:05:36.297739 24878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:44377 every 8 connection(s)
I20260619 11:05:36.298599 24811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:36.299407 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 24811
I20260619 11:05:36.299532 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:36.299856 16628 external_mini_cluster.cc:1428] Setting key 801f2dc815fa931e58daf3fa3e47e501
I20260619 11:05:36.301543 24879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:36.307909 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Bootstrap starting.
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:36.310199 24879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:36.311065 24879 log.cc:824] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:36.312880 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: No bootstrap required, opened a new log
I20260619 11:05:36.314045 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:36.301146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.16.61.62:44377)
0619 11:05:36.301555 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:36.301575 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:36.301617 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:05:36.302401 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:36.302422 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:36.302808 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:36.303154 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:36.303801 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.305233 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.306077 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.306750 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.306908 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:36.309427 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:36.309433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:36.309447 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:36.309476 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:36.311456 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:36.311935 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:36.311941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:36.311946 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:36.312058 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:36.312330 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:36.312333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:36.312334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:36.312687 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:36.312887 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:36.313137 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 11:05:36.313395 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
I20260619 11:05:36.316470 24879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } }
I20260619 11:05:36.316689 24879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:36.316761 24879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1ac342eb0141ffbf37ecb90578b24f, State: Initialized, Role: FOLLOWER
I20260619 11:05:36.317181 24879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } }
I20260619 11:05:36.317322 24879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:36.317404 24879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:36.317495 24879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:36.318380 24879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } }
I20260619 11:05:36.318715 24879 leader_election.cc:302] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c1ac342eb0141ffbf37ecb90578b24f; no voters:
I20260619 11:05:36.319059 24879 leader_election.cc:288] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:36.319139 24884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:36.319365 24884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 1 LEADER]: Becoming Leader. State: Replica: 8c1ac342eb0141ffbf37ecb90578b24f, State: Running, Role: LEADER
I20260619 11:05:36.319723 24884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } }
I20260619 11:05:36.320211 24879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:36.321250 24885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } } }
I20260619 11:05:36.321381 24885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:36.322225 24893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:36.321722 24886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c1ac342eb0141ffbf37ecb90578b24f. Latest consensus state: current_term: 1 leader_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } } }
I20260619 11:05:36.322342 24886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:36.324726 24893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:36.329777 24893 catalog_manager.cc:1347] Generated new cluster ID: 9b59bcb1f35f4a88b2bdf98a8f309a42
I20260619 11:05:36.329876 24893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:36.335966 24893 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:36.337131 24893 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:36.346849 24893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Generated new TSK 0
I20260619 11:05:36.347611 24893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:36.384191 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44377
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:36.490226 24905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:36.490442 24905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:36.490513 24905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:36.494408 24905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:36.494455 24905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:36.494551 24905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:36.499447 24905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:36.500324 24905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:36.501202 24905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:36.509896 24910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:36.509892 24913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:36.509912 24911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:36.510190 24905 server_base.cc:1034] running on GCE node
I20260619 11:05:36.510550 24905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:36.511109 24905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:36.512286 24905 hybrid_clock.cc:648] HybridClock initialized: now 1781867136512260 us; error 43 us; skew 500 ppm
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:36.514915 24905 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:36.515794 24905 webserver.cc:460] Webserver started at http://127.16.61.1:40673/ using document root <none> and password file <none>
I20260619 11:05:36.516294 24905 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:36.516407 24905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:36.516619 24905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:36.518487 24905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "50e025547fe5495d865305019dc29ba7"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "961b62b16513e85ef0f63e6fafe1490f"
server_key_iv: "0e35616a38d2b9500c9eb78dc22259c3"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.519008 24905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "50e025547fe5495d865305019dc29ba7"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "961b62b16513e85ef0f63e6fafe1490f"
server_key_iv: "0e35616a38d2b9500c9eb78dc22259c3"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.522555 24905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:05:36.524741 24920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.525810 24905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:36.525936 24905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "50e025547fe5495d865305019dc29ba7"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "961b62b16513e85ef0f63e6fafe1490f"
server_key_iv: "0e35616a38d2b9500c9eb78dc22259c3"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.526032 24905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:36.535723 24905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:36.538544 24905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:36.538722 24905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:36.539218 24905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:36.539997 24905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:36.540059 24905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.540117 24905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:36.540155 24905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.548735 24905 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:39497
I20260619 11:05:36.548760 25033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:39497 every 8 connection(s)
I20260619 11:05:36.549540 24905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:36.559811 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 24905
I20260619 11:05:36.559954 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:36.560225 16628 external_mini_cluster.cc:1428] Setting key bc31489b4f39c274dadc144585cb6325
I20260619 11:05:36.562075 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:36.551158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:44921 (local address 127.16.61.62:44377)
0619 11:05:36.551285 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:36.551288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:36.552433 (+ 1145us) server_negotiation.cc:407] Connection header received
0619 11:05:36.553251 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:36.553254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:36.553310 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:36.553412 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:36.554485 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.555012 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.556121 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.556348 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.556454 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:36.558630 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:36.558639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:36.558641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:36.558663 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:36.560002 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:36.560546 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:36.560550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:36.560552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:36.560633 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:36.560968 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:36.560971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:36.560972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:36.561195 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:36.561278 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:36.561757 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 11:05:36.561887 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":51}
I20260619 11:05:36.562736 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:36.551497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:44921)
0619 11:05:36.552285 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:36.552321 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:36.553062 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:36.553575 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:36.553582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:36.553901 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:36.554312 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:36.554325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.555133 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.555137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:36.555997 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:36.556005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.556484 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.556488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:36.556644 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:36.557137 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:36.557152 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:36.558479 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:36.560122 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:36.560128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:36.560140 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:36.560450 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:36.560725 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:36.560727 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:36.560730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:36.560875 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:36.561302 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:36.561310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:36.561541 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 11:05:36.561689 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 11:05:36.561903 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":156,"threads_started":1}
I20260619 11:05:36.563947 25034 heartbeater.cc:347] Connected to a master server at 127.16.61.62:44377
I20260619 11:05:36.564167 25034 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:36.564718 25034 heartbeater.cc:511] Master 127.16.61.62:44377 requested a full tablet report, sending...
I20260619 11:05:36.566306 24843 ts_manager.cc:194] Registered new tserver with Master: 50e025547fe5495d865305019dc29ba7 (127.16.61.1:39497)
I20260619 11:05:36.567404 24843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:44921
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:36.586228 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44377
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:36.689599 25039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:36.689864 25039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:36.689940 25039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:36.693840 25039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:36.693919 25039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:36.694012 25039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:36.698978 25039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:36.699959 25039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:36.700881 25039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:36.709836 25045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:36.709857 25039 server_base.cc:1034] running on GCE node
W20260619 11:05:36.709890 25044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:36.709975 25047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:36.710446 25039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:36.711081 25039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:36.712244 25039 hybrid_clock.cc:648] HybridClock initialized: now 1781867136712221 us; error 34 us; skew 500 ppm
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:36.714733 25039 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:36.715451 25039 webserver.cc:460] Webserver started at http://127.16.61.2:39203/ using document root <none> and password file <none>
I20260619 11:05:36.716008 25039 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:36.716069 25039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:36.716219 25039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:36.718029 25039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "ae5df5d99f50425d8f68f4222403e23f"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "59bc610daadd053c341faca7cb454a69"
server_key_iv: "cb2528d738e554ceb5dec8721a950212"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.718523 25039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "ae5df5d99f50425d8f68f4222403e23f"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "59bc610daadd053c341faca7cb454a69"
server_key_iv: "cb2528d738e554ceb5dec8721a950212"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.722164 25039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 11:05:36.724442 25054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.725562 25039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:36.725667 25039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "ae5df5d99f50425d8f68f4222403e23f"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "59bc610daadd053c341faca7cb454a69"
server_key_iv: "cb2528d738e554ceb5dec8721a950212"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.725760 25039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:36.739612 25039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:36.742363 25039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:36.742549 25039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:36.743058 25039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:36.743844 25039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:36.743888 25039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.743943 25039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:36.743980 25039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.752408 25039 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:33017
I20260619 11:05:36.752437 25167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:33017 every 8 connection(s)
I20260619 11:05:36.753185 25039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:36.761430 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 25039
I20260619 11:05:36.761565 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:36.761889 16628 external_mini_cluster.cc:1428] Setting key 73964b2780f72f161e35868de16f6043
I20260619 11:05:36.765708 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:36.754746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:34095 (local address 127.16.61.62:44377)
0619 11:05:36.754878 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:36.754882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:36.755834 (+ 952us) server_negotiation.cc:407] Connection header received
0619 11:05:36.756658 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:36.756662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:36.756716 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:36.756821 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:36.757885 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.758403 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.759494 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.759781 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.759889 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:36.762346 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:36.762349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:36.762352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:36.762381 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:36.763872 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:36.764338 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:36.764342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:36.764344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:36.764423 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:36.764730 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:36.764735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:36.764737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:36.764905 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:36.765005 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:36.765437 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 11:05:36.765547 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:05:36.766475 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:36.755073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:34095)
0619 11:05:36.755656 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:36.755737 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:36.756468 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:36.756989 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:36.756995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:36.757318 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:36.757738 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:36.757751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.758535 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.758538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:36.759376 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:36.759385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.759920 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.759924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:36.760087 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:36.760546 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:36.760561 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:36.762175 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:36.763967 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:36.763972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:36.763985 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:36.764221 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:36.764509 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:36.764513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:36.764515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:36.764641 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:36.765002 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:36.765008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:36.765219 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 11:05:36.765395 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 11:05:36.765673 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":102,"threads_started":1}
I20260619 11:05:36.767527 25168 heartbeater.cc:347] Connected to a master server at 127.16.61.62:44377
I20260619 11:05:36.767773 25168 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:36.768201 25168 heartbeater.cc:511] Master 127.16.61.62:44377 requested a full tablet report, sending...
I20260619 11:05:36.769238 24843 ts_manager.cc:194] Registered new tserver with Master: ae5df5d99f50425d8f68f4222403e23f (127.16.61.2:33017)
I20260619 11:05:36.769995 24843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:34095
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:05:36.787866 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44377
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:36.891651 25173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:36.891891 25173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:36.891937 25173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:36.895802 25173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:36.895852 25173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:36.895934 25173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:36.900925 25173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:36.901821 25173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:36.902737 25173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:36.911473 25178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:36.911582 25173 server_base.cc:1034] running on GCE node
W20260619 11:05:36.911548 25179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:36.911460 25181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:36.912099 25173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:36.912663 25173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:36.913844 25173 hybrid_clock.cc:648] HybridClock initialized: now 1781867136913832 us; error 29 us; skew 500 ppm
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:36.916604 25173 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:36.917433 25173 webserver.cc:460] Webserver started at http://127.16.61.3:45805/ using document root <none> and password file <none>
I20260619 11:05:36.918004 25173 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:36.918068 25173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:36.918296 25173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:36.920476 25173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "7a9634912e6244f1b1858c827ac908d8"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "b9b09170afd676e83c28222c06c24aea"
server_key_iv: "e7b945859a7b605157d78d9aed5e1ee5"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.921070 25173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "7a9634912e6244f1b1858c827ac908d8"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "b9b09170afd676e83c28222c06c24aea"
server_key_iv: "e7b945859a7b605157d78d9aed5e1ee5"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.924831 25173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:05:36.927160 25188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.928352 25173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:36.928473 25173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "7a9634912e6244f1b1858c827ac908d8"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "b9b09170afd676e83c28222c06c24aea"
server_key_iv: "e7b945859a7b605157d78d9aed5e1ee5"
server_key_version: "encryptionkey@0"
I20260619 11:05:36.928570 25173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:36.956318 25173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:36.959276 25173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:36.959466 25173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:36.960012 25173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:36.960744 25173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:36.960815 25173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.960880 25173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:36.960929 25173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:36.969419 25173 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:43811
I20260619 11:05:36.969462 25301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:43811 every 8 connection(s)
I20260619 11:05:36.970264 25173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:36.973076 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 25173
I20260619 11:05:36.973198 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:36.973462 16628 external_mini_cluster.cc:1428] Setting key 939abb5a85fc5cc2160208062ce860c0
Jun 19 11:05:36 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:36.982450 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:36.971884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:34437 (local address 127.16.61.62:44377)
0619 11:05:36.972012 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:36.972016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:36.973055 (+ 1039us) server_negotiation.cc:407] Connection header received
0619 11:05:36.974114 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:36.974117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:36.974168 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:36.974259 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:36.975585 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.976179 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.977289 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.977493 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.977571 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:36.979711 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:36.979716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:36.979718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:36.979745 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:36.980907 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:36.981324 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:36.981326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:36.981328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:36.981399 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:36.981659 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:36.981661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:36.981662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:36.981817 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:36.981888 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:36.982257 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 11:05:36.982342 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:36.983147 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:36.972258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:34437)
0619 11:05:36.972864 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:36.972921 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:36.973898 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:36.974460 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:36.974469 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:36.974904 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:36.975449 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:36.975467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.976272 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.976275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:36.977196 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:36.977205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:36.977610 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:36.977612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:36.977741 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:36.978215 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:36.978230 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:36.979531 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:36.980999 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:36.981003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:36.981013 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:36.981237 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:36.981471 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:36.981473 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:36.981475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:36.981588 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:36.981892 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:36.981896 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:36.982082 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 11:05:36.982220 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 11:05:36.982433 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":103,"threads_started":1}
I20260619 11:05:36.984257 25302 heartbeater.cc:347] Connected to a master server at 127.16.61.62:44377
I20260619 11:05:36.984476 25302 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:36.984858 25302 heartbeater.cc:511] Master 127.16.61.62:44377 requested a full tablet report, sending...
I20260619 11:05:36.985816 24843 ts_manager.cc:194] Registered new tserver with Master: 7a9634912e6244f1b1858c827ac908d8 (127.16.61.3:43811)
I20260619 11:05:36.986423 24843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:34437
I20260619 11:05:36.986994 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab.
Entry for principal oryx/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab.
WARNING: no policy specified for oryx/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab.
Entry for principal oryx/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab.
WARNING: no policy specified for oryx/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab.
Entry for principal oryx/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab.
I20260619 11:05:37.051841 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 24905
I20260619 11:05:37.057683 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 25039
I20260619 11:05:37.062825 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 25173
I20260619 11:05:37.068491 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 24811
I20260619 11:05:37.074298 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44377
--webserver_interface=127.16.61.62
--webserver_port=43341
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:44377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:37.179476 25311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:37.179831 25311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:37.179913 25311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:37.183856 25311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:37.183934 25311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:37.183974 25311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:37.184011 25311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:37.184046 25311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:37.189136 25311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:44377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:44377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=43341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:37.190279 25311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:37.191238 25311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:37.199941 25316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:37.199947 25319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:37.199955 25317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:37.200321 25311 server_base.cc:1034] running on GCE node
I20260619 11:05:37.200984 25311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:37.202426 25311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:37.203612 25311 hybrid_clock.cc:648] HybridClock initialized: now 1781867137203584 us; error 46 us; skew 500 ppm
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867137, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:37.210060 25311 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:37.210920 25311 webserver.cc:460] Webserver started at http://127.16.61.62:43341/ using document root <none> and password file <none>
I20260619 11:05:37.211472 25311 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:37.211546 25311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:37.215488 25311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:37.217562 25326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.218732 25311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:05:37.218868 25311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "8c1ac342eb0141ffbf37ecb90578b24f"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "aa3507e23fd0b93472f0d9d0146dcf2b"
server_key_iv: "2a80e452597b87cb3fb543e6d9115212"
server_key_version: "encryptionkey@0"
I20260619 11:05:37.219406 25311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:37.232806 25311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:37.235504 25311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:37.235743 25311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:37.242157 25311 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:44377
I20260619 11:05:37.242169 25378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:44377 every 8 connection(s)
I20260619 11:05:37.243135 25311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:37.247001 25379 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:05:37.248601 25379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Bootstrap starting.
I20260619 11:05:37.249663 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 25311
I20260619 11:05:37.250108 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:39497
--local_ip_for_outbound_sockets=127.16.61.1
--tserver_master_addrs=127.16.61.62:44377
--webserver_port=40673
--webserver_interface=127.16.61.1
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.61.1
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:05:37.264434 25379 log.cc:824] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:37.270759 25379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:05:37.271118 25379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Bootstrap complete.
I20260619 11:05:37.275134 25379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } }
I20260619 11:05:37.275542 25379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1ac342eb0141ffbf37ecb90578b24f, State: Initialized, Role: FOLLOWER
I20260619 11:05:37.276036 25379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } }
I20260619 11:05:37.276147 25379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:37.276229 25379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:37.276319 25379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:05:37.278115 25379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } }
I20260619 11:05:37.278434 25379 leader_election.cc:302] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c1ac342eb0141ffbf37ecb90578b24f; no voters:
I20260619 11:05:37.278746 25379 leader_election.cc:288] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:05:37.278831 25383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:05:37.279021 25383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [term 2 LEADER]: Becoming Leader. State: Replica: 8c1ac342eb0141ffbf37ecb90578b24f, State: Running, Role: LEADER
I20260619 11:05:37.279290 25383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } }
I20260619 11:05:37.279910 25379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:37.280184 25383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c1ac342eb0141ffbf37ecb90578b24f. Latest consensus state: current_term: 2 leader_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } } }
I20260619 11:05:37.280301 25383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:37.280772 25384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1ac342eb0141ffbf37ecb90578b24f" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 44377 } } }
I20260619 11:05:37.280928 25384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:37.281771 25387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:37.284941 25387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:37.285545 25387 catalog_manager.cc:1259] Loaded cluster ID: 9b59bcb1f35f4a88b2bdf98a8f309a42
I20260619 11:05:37.285602 25387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:37.287894 25387 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:37.288511 25387 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c1ac342eb0141ffbf37ecb90578b24f: Loaded TSK: 0
I20260619 11:05:37.289630 25387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:05:37.359853 25381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:37.360078 25381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:37.360136 25381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:37.364049 25381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:37.364097 25381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:37.364176 25381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:37.369274 25381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.1.keytab
--principal=oryx/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:39497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=40673
--tserver_master_addrs=127.16.61.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:37.370164 25381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:37.371140 25381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:37.381239 25408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:37.381253 25406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:37.381247 25405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:37.381409 25381 server_base.cc:1034] running on GCE node
I20260619 11:05:37.381968 25381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:37.382697 25381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:37.383882 25381 hybrid_clock.cc:648] HybridClock initialized: now 1781867137383860 us; error 39 us; skew 500 ppm
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867137, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:37.387171 25381 init.cc:377] Logged in from keytab as oryx/127.16.61.1@KRBTEST.COM (short username oryx)
I20260619 11:05:37.388254 25381 webserver.cc:460] Webserver started at http://127.16.61.1:40673/ using document root <none> and password file <none>
I20260619 11:05:37.388936 25381 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:37.389041 25381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:37.393816 25381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:37.396317 25415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.397521 25381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:37.397642 25381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "50e025547fe5495d865305019dc29ba7"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "961b62b16513e85ef0f63e6fafe1490f"
server_key_iv: "0e35616a38d2b9500c9eb78dc22259c3"
server_key_version: "encryptionkey@0"
I20260619 11:05:37.398144 25381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:37.409471 25381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:37.412400 25381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:37.412575 25381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:37.413125 25381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:37.413879 25381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:37.413944 25381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.414007 25381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:37.414052 25381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.423329 25381 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:39497
I20260619 11:05:37.423368 25528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:39497 every 8 connection(s)
I20260619 11:05:37.424304 25381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:37.425891 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 25381
I20260619 11:05:37.426383 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:33017
--local_ip_for_outbound_sockets=127.16.61.2
--tserver_master_addrs=127.16.61.62:44377
--webserver_port=39203
--webserver_interface=127.16.61.2
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.61.2
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:05:37.438088 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.426954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:56601 (local address 127.16.61.62:44377)
0619 11:05:37.427474 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.427493 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.427600 (+ 107us) server_negotiation.cc:407] Connection header received
0619 11:05:37.428641 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.428650 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.428923 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.429226 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.430655 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.432069 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.433391 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.433846 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.434020 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.437473 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:56601: BlockingRecv error: recv got EOF from 127.16.61.1:56601 (error 108)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":110,"threads_started":1}
I20260619 11:05:37.438256 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.426677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:56601)
0619 11:05:37.427455 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.427513 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.428270 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.429298 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.429306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.429807 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.430443 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.430465 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.432198 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.432202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.433261 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.433271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.433996 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.434000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.434169 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.434727 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.434742 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.437307 (+ 2565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":135,"threads_started":1}
W20260619 11:05:37.438666 25529 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:44377 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.61.62:44377: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:37.444008 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.439020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:40095)
0619 11:05:37.439126 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.439141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.439210 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.439421 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.439424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.439578 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.439694 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.439704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.440346 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.440349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.441230 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.441238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.441621 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.441624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.441715 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.442122 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.442135 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.443886 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:05:37.444096 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.439117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:40095 (local address 127.16.61.62:44377)
0619 11:05:37.439208 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.439211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.439222 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:05:37.439288 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.439290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.439322 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.439385 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.439793 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.440257 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.441329 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.441528 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.441601 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.443989 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:40095: BlockingRecv error: recv got EOF from 127.16.61.1:40095 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:37.448951 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.444446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:39459)
0619 11:05:37.444645 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.444658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.444732 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.444934 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.444937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.445079 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.445135 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.445140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.445791 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.445797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.446290 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.446296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.446703 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.446707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.446782 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.447193 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.447207 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.448836 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141}
W20260619 11:05:37.449030 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.444543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:39459 (local address 127.16.61.62:44377)
0619 11:05:37.444628 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.444632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.444698 (+ 66us) server_negotiation.cc:407] Connection header received
0619 11:05:37.444801 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.444804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.444847 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.444922 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.445235 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.445693 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.446411 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.446608 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.446704 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.448936 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:39459: BlockingRecv error: recv got EOF from 127.16.61.1:39459 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 11:05:37.449182 25529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:05:37.535558 25533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:37.535859 25533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:37.535902 25533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:37.539806 25533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:37.539853 25533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:37.539907 25533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:37.544809 25533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.2.keytab
--principal=oryx/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:33017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=39203
--tserver_master_addrs=127.16.61.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:37.545668 25533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:37.546547 25533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:37.555308 25541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:37.555393 25539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:37.555449 25538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:37.555721 25533 server_base.cc:1034] running on GCE node
I20260619 11:05:37.556087 25533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:37.556646 25533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:37.557793 25533 hybrid_clock.cc:648] HybridClock initialized: now 1781867137557779 us; error 25 us; skew 500 ppm
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867137, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:37.560300 25533 init.cc:377] Logged in from keytab as oryx/127.16.61.2@KRBTEST.COM (short username oryx)
I20260619 11:05:37.561058 25533 webserver.cc:460] Webserver started at http://127.16.61.2:39203/ using document root <none> and password file <none>
I20260619 11:05:37.561573 25533 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:37.561663 25533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:37.565235 25533 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:37.567209 25548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.568441 25533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:37.568557 25533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "ae5df5d99f50425d8f68f4222403e23f"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "59bc610daadd053c341faca7cb454a69"
server_key_iv: "cb2528d738e554ceb5dec8721a950212"
server_key_version: "encryptionkey@0"
I20260619 11:05:37.569059 25533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:37.583422 25533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:37.586130 25533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:37.586318 25533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:37.586846 25533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:37.587605 25533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:37.587671 25533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.587777 25533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:37.587822 25533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.596452 25533 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:33017
I20260619 11:05:37.596477 25661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:33017 every 8 connection(s)
I20260619 11:05:37.597342 25533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:37.603353 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 25533
I20260619 11:05:37.603868 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:43811
--local_ip_for_outbound_sockets=127.16.61.3
--tserver_master_addrs=127.16.61.62:44377
--webserver_port=45805
--webserver_interface=127.16.61.3
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.61.3
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 11:05:37.608536 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.599005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:34575 (local address 127.16.61.62:44377)
0619 11:05:37.599127 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.599136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.600130 (+ 994us) server_negotiation.cc:407] Connection header received
0619 11:05:37.600977 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.600980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.601039 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.601142 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.602251 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.602757 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.604451 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.604668 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.604751 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.608439 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:34575: BlockingRecv error: recv got EOF from 127.16.61.2:34575 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:05:37.609359 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.599306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:34575)
0619 11:05:37.599990 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.600037 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.600782 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.601305 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.601312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.601629 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.602120 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.602135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.602881 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.602884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.604291 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.604318 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.604811 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.604817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.605048 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.605716 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.605731 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.608336 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":112,"threads_started":1}
W20260619 11:05:37.609794 25662 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:44377 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.61.62:44377: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:37.615048 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.610194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:50197)
0619 11:05:37.610293 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.610308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.610399 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.610652 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.610655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.610800 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.610880 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.610884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.611619 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.611624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.612201 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.612209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.612649 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.612652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.612745 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.613186 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.613198 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.614909 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:05:37.615130 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.610326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:50197 (local address 127.16.61.62:44377)
0619 11:05:37.610473 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.610477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.610489 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:37.610527 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.610528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.610560 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.610639 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.610999 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.611522 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.612315 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.612541 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.612645 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.615017 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:50197: BlockingRecv error: recv got EOF from 127.16.61.2:50197 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:37.620414 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.615492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:48811)
0619 11:05:37.615601 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.615615 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.615704 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.615907 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.615910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.616074 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.616154 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.616159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.616815 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.616818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.617655 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.617664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.618073 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.618076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.618155 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.618615 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.618626 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.620263 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:05:37.620473 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.615583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:48811 (local address 127.16.61.62:44377)
0619 11:05:37.615694 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.615700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.615713 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:37.615785 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.615787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.615816 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.615877 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.616256 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.616725 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.617770 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.617979 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.618052 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.620365 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:48811: BlockingRecv error: recv got EOF from 127.16.61.2:48811 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 11:05:37.620667 25662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:05:37.710856 25665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:37.711123 25665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:37.711189 25665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:37.715176 25665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:37.715252 25665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:37.715353 25665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:37.720427 25665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/krb5kdc/oryx_127.16.61.3.keytab
--principal=oryx/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:43811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=45805
--tserver_master_addrs=127.16.61.62:44377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:37.721316 25665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:37.722249 25665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:37.731022 25673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:37.731072 25671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:37.731118 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:37.731263 25665 server_base.cc:1034] running on GCE node
I20260619 11:05:37.731600 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:37.732208 25665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:37.733374 25665 hybrid_clock.cc:648] HybridClock initialized: now 1781867137733348 us; error 45 us; skew 500 ppm
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867137, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:37.735947 25665 init.cc:377] Logged in from keytab as oryx/127.16.61.3@KRBTEST.COM (short username oryx)
I20260619 11:05:37.736685 25665 webserver.cc:460] Webserver started at http://127.16.61.3:45805/ using document root <none> and password file <none>
I20260619 11:05:37.737192 25665 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:37.737291 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:37.740749 25665 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:37.742599 25680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.743731 25665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:37.743842 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "7a9634912e6244f1b1858c827ac908d8"
format_stamp: "Formatted at 2026-06-19 11:05:36 on dist-test-slave-8vcz"
server_key: "b9b09170afd676e83c28222c06c24aea"
server_key_iv: "e7b945859a7b605157d78d9aed5e1ee5"
server_key_version: "encryptionkey@0"
I20260619 11:05:37.744323 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:37.756860 25665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:37.759804 25665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:37.759981 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:37.760488 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:37.761220 25665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:37.761284 25665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.761344 25665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:37.761390 25665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:37.769666 25665 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:43811
I20260619 11:05:37.769688 25793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:43811 every 8 connection(s)
I20260619 11:05:37.770516 25665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:37.779937 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 25665
W20260619 11:05:37.781144 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.772209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:52527 (local address 127.16.61.62:44377)
0619 11:05:37.772369 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.772372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.773315 (+ 943us) server_negotiation.cc:407] Connection header received
0619 11:05:37.774071 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.774073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.774116 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.774189 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.775216 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.775777 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.776847 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.777128 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.777218 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.781016 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:52527: BlockingRecv error: recv got EOF from 127.16.61.3:52527 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 11:05:37.781761 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.772517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:52527)
0619 11:05:37.773187 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.773223 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.773909 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.774329 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.774335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.774670 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.775077 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.775090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.775878 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.775881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.776723 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.776731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.777320 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.777324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.777461 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.778049 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.778067 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.780828 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":131,"threads_started":1}
W20260619 11:05:37.782258 25794 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:44377 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.61.62:44377: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:37.787379 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.782657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:53805)
0619 11:05:37.782828 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.782840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.782903 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.783084 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.783085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.783248 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.783352 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.783358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.784107 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.784110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.784787 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.784795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.785225 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.785228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.785358 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.785738 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.785755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.787295 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 11:05:37.787550 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.782696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:53805 (local address 127.16.61.62:44377)
0619 11:05:37.782785 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.782789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.782899 (+ 110us) server_negotiation.cc:407] Connection header received
0619 11:05:37.782975 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.782977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.783014 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.783080 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.783455 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.783985 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.784914 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.785117 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.785192 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.787435 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:53805: BlockingRecv error: recv got EOF from 127.16.61.3:53805 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 11:05:37.789702 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.780957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.16.61.62:44377)
0619 11:05:37.781197 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.781209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.781223 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:05:37.781363 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.781366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.781410 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.781503 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.782376 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.783080 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.784072 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.784313 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.784385 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.785825 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:37.785830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:37.785842 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:37.785866 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:37.787713 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:37.788192 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:37.788198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:37.788203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:37.788313 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:37.788586 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:37.788590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:37.788592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:37.788942 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:37.789126 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:37.789391 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 11:05:37.789597 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:37 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:37.793038 25799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:37.787808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:54197)
0619 11:05:37.788064 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:37.788075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:37.788174 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:37.788377 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:37.788379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:37.788553 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:37.788630 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.788635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.789283 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.789288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.790047 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:37.790057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.790584 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.790587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:37.790696 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.791096 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:37.791109 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:37.792935 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
W20260619 11:05:37.793092 25532 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:37.787847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:54197 (local address 127.16.61.62:44377)
0619 11:05:37.787977 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:37.787981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:37.788139 (+ 158us) server_negotiation.cc:407] Connection header received
0619 11:05:37.788247 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:37.788250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:37.788284 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:37.788365 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:37.788731 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.789185 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.790148 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:37.790464 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:37.790546 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:37.793027 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:54197: BlockingRecv error: recv got EOF from 127.16.61.3:54197 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 11:05:37.793325 25794 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 11:05:38 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:38 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:38.454710 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:38.449667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:48929)
0619 11:05:38.449896 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:38.449909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:38.449992 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:38.450209 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:38.450211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:38.450369 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:38.450458 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:38.450463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.451135 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.451137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:38.451744 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:38.451753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.452200 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.452204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:38.452289 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:38.452905 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:38.452919 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:38.454597 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
W20260619 11:05:38.454900 25801 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:38.449739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:48929 (local address 127.16.61.62:44377)
0619 11:05:38.449965 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:38.449968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:38.449980 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:38.450068 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:38.450070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:38.450110 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:38.450186 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:38.450542 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.451052 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.451852 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.452095 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.452197 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:38.454710 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:48929: BlockingRecv error: recv got EOF from 127.16.61.1:48929 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
Jun 19 11:05:38 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:38 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:38.626477 25802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:38.621305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:56967)
0619 11:05:38.621558 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:38.621571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:38.621658 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:38.621868 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:38.621870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:38.622171 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:38.622296 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:38.622302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.623006 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.623009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:38.623701 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:38.623710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.624108 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.624110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:38.624175 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:38.624634 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:38.624645 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:38.626344 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":91,"threads_started":1}
W20260619 11:05:38.626698 25801 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:38.621250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:56967 (local address 127.16.61.62:44377)
0619 11:05:38.621424 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:38.621431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:38.621636 (+ 205us) server_negotiation.cc:407] Connection header received
0619 11:05:38.621731 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:38.621734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:38.621774 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:38.621852 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:38.622407 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.622911 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.623806 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.624023 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.624097 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:38.626483 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:56967: BlockingRecv error: recv got EOF from 127.16.61.2:56967 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 11:05:38 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:38 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:38.798986 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:38.793858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:58069)
0619 11:05:38.794106 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:38.794119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:38.794205 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:38.794427 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:38.794429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:38.794589 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:38.794694 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:38.794699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.795388 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.795390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:38.795969 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:38.795976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.796357 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.796359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:38.796441 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:38.796877 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:38.796887 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:38.798860 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
W20260619 11:05:38.799125 25801 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:38.793955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:58069 (local address 127.16.61.62:44377)
0619 11:05:38.794077 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:38.794080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:38.794179 (+ 99us) server_negotiation.cc:407] Connection header received
0619 11:05:38.794277 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:38.794279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:38.794321 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:38.794391 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:38.794787 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.795299 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.796064 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:38.796275 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:38.796347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:38.798977 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:58069: BlockingRecv error: recv got EOF from 127.16.61.3:58069 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 11:05:39 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:39 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:39.460817 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:39.455519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:49811)
0619 11:05:39.455798 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:39.455814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:39.455935 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:39.456240 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:39.456243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:39.456421 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:39.456514 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:39.456519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.457240 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.457243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:39.457817 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:39.457825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.458294 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.458297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:39.458360 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:39.458831 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:39.458842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:39.460683 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":95,"threads_started":1}
W20260619 11:05:39.460985 25805 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:39.455545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:49811 (local address 127.16.61.62:44377)
0619 11:05:39.455819 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:39.455823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:39.455864 (+ 41us) server_negotiation.cc:407] Connection header received
0619 11:05:39.456068 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:39.456073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:39.456134 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:39.456223 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:39.456614 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.457121 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.457952 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.458176 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.458257 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:39.460816 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:49811: BlockingRecv error: recv got EOF from 127.16.61.1:49811 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":67,"threads_started":1}
Jun 19 11:05:39 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:39 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:39.632782 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:39.627332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:57585)
0619 11:05:39.627621 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:39.627633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:39.627802 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:39.628087 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:39.628090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:39.628305 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:39.628394 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:39.628399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.629147 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.629151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:39.629704 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:39.629711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.630169 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.630172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:39.630259 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:39.630759 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:39.630771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:39.632595 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
W20260619 11:05:39.632819 25805 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:39.627406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:57585 (local address 127.16.61.62:44377)
0619 11:05:39.627551 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:39.627556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:39.627735 (+ 179us) server_negotiation.cc:407] Connection header received
0619 11:05:39.627929 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:39.627932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:39.627984 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:39.628069 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:39.628508 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.629030 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.629826 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.630059 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.630168 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:39.632722 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:57585: BlockingRecv error: recv got EOF from 127.16.61.2:57585 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 11:05:39 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:39 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:39.804889 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:39.799638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:55949)
0619 11:05:39.799924 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:39.799937 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:39.800021 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:39.800252 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:39.800275 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:39.800438 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:39.800531 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:39.800536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.801345 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.801348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:39.801977 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:39.801989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.802423 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.802427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:39.802492 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:39.803030 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:39.803041 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:39.804781 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":77,"threads_started":1}
W20260619 11:05:39.805073 25805 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:39.799816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:55949 (local address 127.16.61.62:44377)
0619 11:05:39.799961 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:39.799965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:39.799980 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:05:39.800103 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:39.800105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:39.800147 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:39.800211 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:39.800625 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.801209 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.802087 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:39.802312 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:39.802389 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:39.804942 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:55949: BlockingRecv error: recv got EOF from 127.16.61.3:55949 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 11:05:40 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:40 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:40.466703 25808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:40.461484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:58841)
0619 11:05:40.461740 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:40.461753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:40.461834 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:40.462048 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:40.462050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:40.462214 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:40.462306 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:40.462311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.463001 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.463004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:40.463552 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:40.463559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.463963 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.463965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:40.464022 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:40.464493 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:40.464504 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:40.466577 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
W20260619 11:05:40.466821 25809 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:40.461598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:58841 (local address 127.16.61.62:44377)
0619 11:05:40.461826 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:40.461829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:40.461841 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:40.461898 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:40.461901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:40.461938 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:40.462012 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:40.462397 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.462915 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.463642 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.463875 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.463949 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:40.466701 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:58841: BlockingRecv error: recv got EOF from 127.16.61.1:58841 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
Jun 19 11:05:40 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:40 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:40.639154 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:40.633558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:53565)
0619 11:05:40.633812 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:40.633825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:40.633960 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:40.634333 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:40.634336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:40.634586 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:40.634706 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:40.634713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.635627 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.635630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:40.636324 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:40.636333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.636744 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.636747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:40.636805 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:40.637265 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:40.637276 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:40.639041 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
W20260619 11:05:40.639317 25809 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:40.633615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:53565 (local address 127.16.61.62:44377)
0619 11:05:40.633757 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:40.633761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:40.633893 (+ 132us) server_negotiation.cc:407] Connection header received
0619 11:05:40.634070 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:40.634090 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:40.634158 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:40.634259 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:40.634843 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.635530 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.636421 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.636649 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.636728 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:40.639167 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:53565: BlockingRecv error: recv got EOF from 127.16.61.2:53565 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 11:05:40 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:40 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:40.810956 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:40.805601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:45999)
0619 11:05:40.805876 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:40.805888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:40.806023 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:40.806281 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:40.806284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:40.806479 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:40.806567 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:40.806572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.807306 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.807313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:40.807925 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:40.807933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.808380 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.808384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:40.808458 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:40.808995 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:40.809006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:40.810785 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":95,"threads_started":1}
W20260619 11:05:40.810988 25809 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:40.805649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:45999 (local address 127.16.61.62:44377)
0619 11:05:40.805801 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:40.805805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:40.805964 (+ 159us) server_negotiation.cc:407] Connection header received
0619 11:05:40.806118 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:40.806122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:40.806175 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:40.806261 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:40.806678 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.807189 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.808051 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:40.808269 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:40.808377 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:40.810890 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:45999: BlockingRecv error: recv got EOF from 127.16.61.3:45999 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 11:05:41 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:41 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:41.473012 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:41.467534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:47889)
0619 11:05:41.467922 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:41.467939 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:41.468084 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:41.468384 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:41.468387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:41.468624 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:41.468748 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:41.468755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.469461 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.469463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:41.470008 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:41.470015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.470399 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.470401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:41.470460 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:41.470993 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:41.471014 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:41.472840 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":110,"threads_started":1}
W20260619 11:05:41.473066 25813 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:41.467594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:47889 (local address 127.16.61.62:44377)
0619 11:05:41.467899 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:41.467904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:41.468038 (+ 134us) server_negotiation.cc:407] Connection header received
0619 11:05:41.468185 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:41.468189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:41.468245 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:41.468331 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:41.468857 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.469366 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.470096 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.470309 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.470379 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:41.472953 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:47889: BlockingRecv error: recv got EOF from 127.16.61.1:47889 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
Jun 19 11:05:41 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:41 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:41.644886 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:41.639924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:42915)
0619 11:05:41.640185 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:41.640200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:41.640300 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:41.640516 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:41.640518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:41.640689 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:41.640780 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:41.640785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.641472 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.641474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:41.642022 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:41.642029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.642418 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.642421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:41.642477 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:41.643024 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:41.643037 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:41.644775 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
W20260619 11:05:41.645010 25813 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:41.640022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:42915 (local address 127.16.61.62:44377)
0619 11:05:41.640153 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:41.640156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:41.640257 (+ 101us) server_negotiation.cc:407] Connection header received
0619 11:05:41.640364 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:41.640366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:41.640404 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:41.640472 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:41.640875 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.641384 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.642120 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.642329 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.642400 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:41.644891 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:42915: BlockingRecv error: recv got EOF from 127.16.61.2:42915 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 11:05:41 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:41 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:41.817000 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:41.811896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:50839)
0619 11:05:41.812156 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:41.812168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:41.812251 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:41.812472 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:41.812474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:41.812638 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:41.812725 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:41.812729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.813399 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.813401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:41.813947 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:41.813953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.814341 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.814343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:41.814402 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:41.814900 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:41.814912 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:41.816812 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":99,"threads_started":1}
W20260619 11:05:41.817027 25813 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:41.812078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:50839 (local address 127.16.61.62:44377)
0619 11:05:41.812198 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:41.812201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:41.812213 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:41.812331 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:41.812333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:41.812375 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:41.812456 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:41.812815 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.813313 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.814050 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:41.814254 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:41.814325 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:41.816930 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:50839: BlockingRecv error: recv got EOF from 127.16.61.3:50839 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 11:05:42 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:42 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:42.483208 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:42.473777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:55011)
0619 11:05:42.474054 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:42.474067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:42.474153 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:42.474402 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:42.474404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:42.474588 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:42.474676 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:42.474681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.475729 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.475732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:42.476371 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:42.476380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.476802 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.476805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:42.476867 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:42.481242 (+ 4375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:42.481255 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:42.483040 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
W20260619 11:05:42.483281 25817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:42.473890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:55011 (local address 127.16.61.62:44377)
0619 11:05:42.474116 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:42.474119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:42.474133 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:05:42.474246 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:42.474248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:42.474290 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:42.474365 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:42.474792 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.475557 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.476471 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.476692 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.476771 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:42.483158 (+ 6387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:55011: BlockingRecv error: recv got EOF from 127.16.61.1:55011 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":50,"threads_started":1}
Jun 19 11:05:42 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:42 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:42.651454 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:42.645823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:45165)
0619 11:05:42.646099 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:42.646114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:42.646222 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:42.646492 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:42.646495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:42.646779 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:42.646924 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:42.646932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.647796 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.647800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:42.648399 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:42.648407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.648843 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.648847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:42.648932 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:42.649457 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:42.649469 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:42.651322 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
W20260619 11:05:42.651577 25817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:42.645959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:45165 (local address 127.16.61.62:44377)
0619 11:05:42.646094 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:42.646100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:42.646204 (+ 104us) server_negotiation.cc:407] Connection header received
0619 11:05:42.646327 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:42.646329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:42.646379 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:42.646467 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:42.647033 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.647637 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.648516 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.648735 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.648841 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:42.651456 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:45165: BlockingRecv error: recv got EOF from 127.16.61.2:45165 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 11:05:42 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:42 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:42.823808 25819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:42.817939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:60493)
0619 11:05:42.818181 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:42.818201 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:42.818318 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:42.818592 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:42.818595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:42.818806 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:42.818905 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:42.818910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.819639 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.819644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:42.820243 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:42.820251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.820705 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.820709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:42.820809 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:42.821554 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:42.821566 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:42.823625 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
W20260619 11:05:42.823881 25817 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:42.817975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:60493 (local address 127.16.61.62:44377)
0619 11:05:42.818106 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:42.818111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:42.818269 (+ 158us) server_negotiation.cc:407] Connection header received
0619 11:05:42.818416 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:42.818419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:42.818488 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:42.818575 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:42.819016 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.819536 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.820359 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:42.820591 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:42.820693 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:42.823761 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:60493: BlockingRecv error: recv got EOF from 127.16.61.3:60493 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 11:05:43 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:43 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:43.489190 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:43.483972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:49611)
0619 11:05:43.484258 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:43.484272 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:43.484367 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:43.484580 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:43.484582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:43.484756 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:43.484848 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:43.484853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.485547 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.485550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:43.486102 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:43.486109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.486506 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.486509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:43.486567 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:43.487035 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:43.487046 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:43.489083 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
W20260619 11:05:43.489341 25821 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:43.484018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:49611 (local address 127.16.61.62:44377)
0619 11:05:43.484259 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:43.484262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:43.484344 (+ 82us) server_negotiation.cc:407] Connection header received
0619 11:05:43.484434 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:43.484436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:43.484476 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:43.484549 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:43.484950 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.485462 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.486200 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.486411 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.486488 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:43.489204 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:49611: BlockingRecv error: recv got EOF from 127.16.61.1:49611 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
Jun 19 11:05:43 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:43 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:43.657428 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:43.652363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:55545)
0619 11:05:43.652611 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:43.652626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:43.652729 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:43.652949 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:43.652951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:43.653136 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:43.653227 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:43.653232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.653920 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.653922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:43.654463 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:43.654471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.654848 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.654850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:43.654910 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:43.655383 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:43.655394 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:43.657302 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
W20260619 11:05:43.657627 25821 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:43.652423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:55545 (local address 127.16.61.62:44377)
0619 11:05:43.652578 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:43.652582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:43.652703 (+ 121us) server_negotiation.cc:407] Connection header received
0619 11:05:43.652805 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:43.652807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:43.652847 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:43.652916 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:43.653323 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.653835 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.654557 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.654760 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.654831 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:43.657493 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:55545: BlockingRecv error: recv got EOF from 127.16.61.2:55545 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 11:05:43 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:43 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:43.830224 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:43.824779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:41651)
0619 11:05:43.825069 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:43.825081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:43.825201 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:43.825473 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:43.825476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:43.825677 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:43.825764 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:43.825769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.826488 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.826492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:43.827052 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:43.827059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.827508 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.827512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:43.827596 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:43.828123 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:43.828134 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:43.830115 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
W20260619 11:05:43.830349 25821 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:43.824839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:41651 (local address 127.16.61.62:44377)
0619 11:05:43.824989 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:43.824994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:43.825156 (+ 162us) server_negotiation.cc:407] Connection header received
0619 11:05:43.825305 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:43.825309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:43.825360 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:43.825452 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:43.825878 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.826383 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.827167 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:43.827398 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:43.827513 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:43.830237 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:41651: BlockingRecv error: recv got EOF from 127.16.61.3:41651 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 11:05:44 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:44 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:44.494802 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:44.489909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:45051)
0619 11:05:44.490213 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:44.490227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:44.490322 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:44.490548 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:44.490550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:44.490727 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:44.490817 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:44.490823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.491503 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.491506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:44.492067 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:44.492074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.492481 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.492483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:44.492538 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:44.493009 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:44.493020 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:44.494689 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":79,"threads_started":1}
W20260619 11:05:44.494969 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:44.489968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:45051 (local address 127.16.61.62:44377)
0619 11:05:44.490217 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:44.490221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:44.490295 (+ 74us) server_negotiation.cc:407] Connection header received
0619 11:05:44.490397 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:44.490399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:44.490441 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:44.490515 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:44.490908 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.491419 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.492163 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.492390 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.492469 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:44.494838 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:45051: BlockingRecv error: recv got EOF from 127.16.61.1:45051 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
Jun 19 11:05:44 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:44 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:44.663050 25826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:44.658204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:39479)
0619 11:05:44.658519 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:44.658533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:44.658634 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:44.658848 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:44.658850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:44.659029 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:44.659122 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:44.659127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.659858 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.659861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:44.660410 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:44.660418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.660799 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.660801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:44.660858 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:44.661317 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:44.661328 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:44.662940 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
W20260619 11:05:44.663210 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:44.658213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:39479 (local address 127.16.61.62:44377)
0619 11:05:44.658349 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:44.658352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:44.658605 (+ 253us) server_negotiation.cc:407] Connection header received
0619 11:05:44.658700 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:44.658702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:44.658743 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:44.658810 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:44.659214 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.659768 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.660504 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.660710 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.660781 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:44.663063 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:39479: BlockingRecv error: recv got EOF from 127.16.61.2:39479 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 11:05:44 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:44 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:44.836323 25827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:44.831011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:56893)
0619 11:05:44.831315 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:44.831328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:44.831421 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:44.831645 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:44.831647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:44.831943 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:44.832042 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:44.832048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.832765 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.832767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:44.833297 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:44.833303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.833730 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.833732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:44.833790 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:44.834320 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:44.834331 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:44.836212 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
W20260619 11:05:44.836449 25825 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:44.831072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:56893 (local address 127.16.61.62:44377)
0619 11:05:44.831194 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:44.831197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:44.831393 (+ 196us) server_negotiation.cc:407] Connection header received
0619 11:05:44.831498 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:44.831500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:44.831543 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:44.831615 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:44.832147 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.832676 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.833399 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:44.833633 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:44.833718 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:44.836337 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:56893: BlockingRecv error: recv got EOF from 127.16.61.3:56893 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 11:05:45 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:45 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:45.501019 25828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:45.495561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:38393)
0619 11:05:45.495824 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:45.495839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:45.495942 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:45.496259 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:45.496261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:45.496503 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:45.496601 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:45.496608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.497505 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.497508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:45.498070 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:45.498078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.498509 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.498512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:45.498569 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:45.499055 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:45.499066 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:45.500896 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":130,"threads_started":1}
W20260619 11:05:45.501157 25829 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:45.495739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:38393 (local address 127.16.61.62:44377)
0619 11:05:45.495985 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:45.495989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:45.496009 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:05:45.496051 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:45.496057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:45.496114 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:45.496231 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:45.496720 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.497388 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.498183 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.498416 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.498494 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:45.501029 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:38393: BlockingRecv error: recv got EOF from 127.16.61.1:38393 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
Jun 19 11:05:45 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:45 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:45.669216 25830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:45.663863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:53541)
0619 11:05:45.664120 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:45.664132 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:45.664222 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:45.664453 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:45.664455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:45.664634 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:45.664724 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:45.664729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.665538 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.665540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:45.666113 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:45.666122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.666624 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.666627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:45.666716 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:45.667289 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:45.667302 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:45.669105 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
W20260619 11:05:45.669364 25829 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:45.663978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:53541 (local address 127.16.61.62:44377)
0619 11:05:45.664111 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:45.664115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:45.664197 (+ 82us) server_negotiation.cc:407] Connection header received
0619 11:05:45.664298 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:45.664300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:45.664346 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:45.664419 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:45.664825 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.665446 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.666232 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.666513 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.666614 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:45.669225 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:53541: BlockingRecv error: recv got EOF from 127.16.61.2:53541 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 11:05:45 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:45 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:45.842319 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:45.837282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:52769)
0619 11:05:45.837570 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:45.837586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:45.837701 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:45.838019 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:45.838022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:45.838271 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:45.838390 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:45.838397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.839093 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.839095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:45.839633 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:45.839639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.840072 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.840074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:45.840133 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:45.840586 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:45.840596 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:45.842211 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
W20260619 11:05:45.842562 25829 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:45.837319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:52769 (local address 127.16.61.62:44377)
0619 11:05:45.837455 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:45.837460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:45.837663 (+ 203us) server_negotiation.cc:407] Connection header received
0619 11:05:45.837824 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:45.837830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:45.837892 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:45.837966 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:45.838498 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.839006 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.839755 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:45.839979 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:45.840055 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:45.842336 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:52769: BlockingRecv error: recv got EOF from 127.16.61.3:52769 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 11:05:46 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:46 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:46.506783 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:46.501841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:51115)
0619 11:05:46.502105 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:46.502118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:46.502212 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:46.502435 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:46.502437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:46.502609 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:46.502702 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:46.502708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.503457 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.503459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:46.504030 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:46.504037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.504418 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.504420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:46.504477 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:46.504925 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:46.504936 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:46.506623 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
W20260619 11:05:46.506886 25833 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:46.501879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:51115 (local address 127.16.61.62:44377)
0619 11:05:46.502128 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:46.502131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:46.502167 (+ 36us) server_negotiation.cc:407] Connection header received
0619 11:05:46.502290 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:46.502292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:46.502333 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:46.502406 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:46.502793 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.503369 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.504119 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.504331 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.504404 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:46.506754 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:51115: BlockingRecv error: recv got EOF from 127.16.61.1:51115 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
Jun 19 11:05:46 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:46 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:46.675122 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:46.669924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:41935)
0619 11:05:46.670171 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:46.670185 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:46.670276 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:46.670503 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:46.670505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:46.670675 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:46.670766 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:46.670771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.671492 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.671494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:46.672066 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:46.672074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.672535 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.672538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:46.672601 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:46.673053 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:46.673064 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:46.674955 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":87,"threads_started":1}
W20260619 11:05:46.675156 25833 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:46.670036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:41935 (local address 127.16.61.62:44377)
0619 11:05:46.670167 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:46.670171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:46.670251 (+ 80us) server_negotiation.cc:407] Connection header received
0619 11:05:46.670354 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:46.670356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:46.670399 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:46.670471 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:46.670878 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.671394 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.672185 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.672436 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.672523 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:46.675063 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:41935: BlockingRecv error: recv got EOF from 127.16.61.2:41935 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 11:05:46 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:46 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:46.849001 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:46.843020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:50745)
0619 11:05:46.843312 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:46.843326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:46.843458 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:46.843764 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:46.843767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:46.843980 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:46.844067 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:46.844072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.844799 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.844812 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:46.845377 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:46.845384 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.845828 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.845831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:46.845913 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:46.846465 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:46.846476 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:46.848825 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
W20260619 11:05:46.849038 25833 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:46.843087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:50745 (local address 127.16.61.62:44377)
0619 11:05:46.843248 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:46.843253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:46.843405 (+ 152us) server_negotiation.cc:407] Connection header received
0619 11:05:46.843561 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:46.843565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:46.843616 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:46.843746 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:46.844176 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.844694 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.845489 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:46.845713 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:46.845822 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:46.848944 (+ 3122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:50745: BlockingRecv error: recv got EOF from 127.16.61.3:50745 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 11:05:47 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:47 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:47.516677 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:47.507635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:40177)
0619 11:05:47.507909 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:47.507923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:47.508008 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:47.508238 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:47.508239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:47.508408 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:47.508503 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:47.508508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.509295 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.509298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:47.509854 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:47.509861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.510263 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.510265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:47.510323 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:47.514375 (+ 4052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:47.514413 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:47.516475 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":65,"threads_started":1}
W20260619 11:05:47.516808 25837 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:47.507734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:40177 (local address 127.16.61.62:44377)
0619 11:05:47.507972 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:47.507976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:47.507988 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:05:47.508090 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:47.508092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:47.508133 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:47.508207 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:47.508623 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.509189 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.509957 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.510173 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.510251 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:47.516668 (+ 6417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:40177: BlockingRecv error: recv got EOF from 127.16.61.1:40177 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
Jun 19 11:05:47 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:47 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:47.681133 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:47.675873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:40741)
0619 11:05:47.676128 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:47.676141 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:47.676227 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:47.676456 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:47.676458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:47.676633 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:47.676723 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:47.676727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.677406 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.677408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:47.677961 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:47.677968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.678377 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.678379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:47.678450 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:47.678930 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:47.678942 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:47.681012 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
W20260619 11:05:47.681197 25837 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:47.675950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:40741 (local address 127.16.61.62:44377)
0619 11:05:47.676092 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:47.676095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:47.676189 (+ 94us) server_negotiation.cc:407] Connection header received
0619 11:05:47.676305 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:47.676307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:47.676347 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:47.676409 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:47.676814 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.677321 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.678068 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.678283 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.678366 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:47.681109 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:40741: BlockingRecv error: recv got EOF from 127.16.61.2:40741 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 11:05:47 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:47 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:47.854632 25839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:47.849706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:52129)
0619 11:05:47.849973 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:47.849986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:47.850071 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:47.850354 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:47.850356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:47.850529 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:47.850621 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:47.850626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.851306 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.851308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:47.851878 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:47.851885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.852267 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.852270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:47.852324 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:47.852783 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:47.852793 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:47.854448 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
W20260619 11:05:47.854648 25837 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:47.849864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:52129 (local address 127.16.61.62:44377)
0619 11:05:47.849995 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:47.849999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:47.850034 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:05:47.850198 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:47.850202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:47.850252 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:47.850329 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:47.850715 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.851223 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.851973 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:47.852179 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:47.852253 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:47.854563 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:52129: BlockingRecv error: recv got EOF from 127.16.61.3:52129 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 11:05:48 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:48 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:48.522614 25840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:48.517355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:55805)
0619 11:05:48.517633 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:48.517655 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:48.517747 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:48.517984 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:48.517986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:48.518160 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:48.518300 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:48.518306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.519018 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.519020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:48.519592 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:48.519600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.520048 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.520051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:48.520118 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:48.520629 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:48.520640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:48.522499 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
W20260619 11:05:48.522771 25841 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:48.517492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:55805 (local address 127.16.61.62:44377)
0619 11:05:48.517697 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:48.517700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:48.517713 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:48.517832 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:48.517834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:48.517875 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:48.517956 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:48.518395 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.518924 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.519702 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.519915 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.519998 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:48.522639 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:55805: BlockingRecv error: recv got EOF from 127.16.61.1:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
Jun 19 11:05:48 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:48 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:48.687314 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:48.681961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:40841)
0619 11:05:48.682209 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:48.682222 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:48.682321 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:48.682558 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:48.682560 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:48.682749 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:48.682839 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:48.682844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.683536 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.683538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:48.684160 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:48.684169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.684709 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.684713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:48.684803 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:48.685360 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:48.685373 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:48.687112 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":64,"threads_started":1}
W20260619 11:05:48.687350 25841 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:48.682075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:40841 (local address 127.16.61.62:44377)
0619 11:05:48.682228 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:48.682231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:48.682291 (+ 60us) server_negotiation.cc:407] Connection header received
0619 11:05:48.682398 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:48.682400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:48.682457 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:48.682521 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:48.682935 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.683444 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.684289 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.684568 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.684705 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:48.687243 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:40841: BlockingRecv error: recv got EOF from 127.16.61.2:40841 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 11:05:48 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:48 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:48.860392 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:48.855428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:51987)
0619 11:05:48.855723 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:48.855736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:48.855828 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:48.856058 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:48.856060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:48.856228 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:48.856318 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:48.856323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.857010 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.857012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:48.857556 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:48.857562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.857940 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.857943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:48.858008 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:48.858537 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:48.858549 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:48.860282 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
W20260619 11:05:48.860562 25841 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:48.855494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:51987 (local address 127.16.61.62:44377)
0619 11:05:48.855610 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:48.855613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:48.855803 (+ 190us) server_negotiation.cc:407] Connection header received
0619 11:05:48.855902 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:48.855904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:48.855955 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:48.856021 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:48.856408 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.856906 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.857647 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:48.857853 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:48.857925 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:48.860397 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:51987: BlockingRecv error: recv got EOF from 127.16.61.3:51987 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 11:05:49 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:49 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:49.528703 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:49.523378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:60671)
0619 11:05:49.523706 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:49.523737 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:49.523838 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:49.524126 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:49.524129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:49.524332 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:49.524421 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:49.524427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.525187 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.525192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:49.525745 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:49.525752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.526221 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.526225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:49.526311 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:49.526770 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:49.526781 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:49.528528 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":73,"threads_started":1}
W20260619 11:05:49.528813 25845 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:49.523407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:60671 (local address 127.16.61.62:44377)
0619 11:05:49.523777 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:49.523782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:49.523798 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:05:49.523944 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:49.523947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:49.524011 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:49.524103 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:49.524542 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.525078 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.525860 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.526111 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.526222 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:49.528658 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:60671: BlockingRecv error: recv got EOF from 127.16.61.1:60671 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
Jun 19 11:05:49 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:49 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:49.693233 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:49.688078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:57743)
0619 11:05:49.688348 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:49.688362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:49.688464 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:49.688701 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:49.688704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:49.688893 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:49.688986 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:49.688991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.689681 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.689683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:49.690239 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:49.690247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.690642 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.690644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:49.690705 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:49.691177 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:49.691189 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:49.693068 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
W20260619 11:05:49.693286 25845 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:49.688126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:57743 (local address 127.16.61.62:44377)
0619 11:05:49.688276 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:49.688280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:49.688437 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:05:49.688551 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:49.688553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:49.688597 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:49.688663 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:49.689080 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.689583 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.690337 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.690549 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.690622 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:49.693182 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:57743: BlockingRecv error: recv got EOF from 127.16.61.2:57743 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 11:05:49 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:49 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:49.866477 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:49.861142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:56059)
0619 11:05:49.861494 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:49.861507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:49.861598 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:49.861815 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:49.861818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:49.861996 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:49.862087 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:49.862092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.862928 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.862931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:49.863628 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:49.863636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.864076 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.864079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:49.864133 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:49.864600 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:49.864611 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:49.866313 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":188,"threads_started":1}
W20260619 11:05:49.866533 25845 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:49.861233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:56059 (local address 127.16.61.62:44377)
0619 11:05:49.861368 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:49.861371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:49.861558 (+ 187us) server_negotiation.cc:407] Connection header received
0619 11:05:49.861662 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:49.861664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:49.861704 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:49.861773 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:49.862182 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.862808 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.863758 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:49.863982 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:49.864065 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:49.866426 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:56059: BlockingRecv error: recv got EOF from 127.16.61.3:56059 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 11:05:50 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:50 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:50.535264 25848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:50.529476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:48925)
0619 11:05:50.529748 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:50.529761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:50.529847 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:50.530074 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:50.530077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:50.530252 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:50.530378 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:50.530384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.531146 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.531148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:50.531804 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:50.531814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.532316 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.532319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:50.532387 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:50.533026 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:50.533038 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:50.535154 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":97,"threads_started":1}
W20260619 11:05:50.535498 25849 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:50.529571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:48925 (local address 127.16.61.62:44377)
0619 11:05:50.529801 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:50.529811 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:50.529824 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:50.529929 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:50.529931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:50.529971 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:50.530041 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:50.530507 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.531041 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.531922 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.532146 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.532237 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:50.535272 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:48925: BlockingRecv error: recv got EOF from 127.16.61.1:48925 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
Jun 19 11:05:50 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:50 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:50.699384 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:50.693997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:45783)
0619 11:05:50.694315 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:50.694327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:50.694462 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:50.694766 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:50.694768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:50.694943 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:50.695046 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:50.695051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.695750 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.695752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:50.696434 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:50.696443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.696867 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.696870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:50.696924 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:50.697488 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:50.697500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:50.699269 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":99,"threads_started":1}
W20260619 11:05:50.699517 25849 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:50.694103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:45783 (local address 127.16.61.62:44377)
0619 11:05:50.694228 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:50.694232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:50.694421 (+ 189us) server_negotiation.cc:407] Connection header received
0619 11:05:50.694563 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:50.694566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:50.694626 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:50.694731 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:50.695139 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.695653 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.696534 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.696778 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.696852 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:50.699393 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:45783: BlockingRecv error: recv got EOF from 127.16.61.2:45783 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 19 11:05:50 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:50 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:50.872498 25851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:50.867295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:34543)
0619 11:05:50.867554 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:50.867567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:50.867676 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:50.867900 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:50.867902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:50.868073 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:50.868168 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:50.868173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.868866 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.868868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:50.869421 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:50.869427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.869824 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.869826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:50.869886 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:50.870414 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:50.870425 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:50.872385 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
W20260619 11:05:50.872684 25849 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:50.867350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:34543 (local address 127.16.61.62:44377)
0619 11:05:50.867493 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:50.867496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:50.867631 (+ 135us) server_negotiation.cc:407] Connection header received
0619 11:05:50.867752 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:50.867754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:50.867797 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:50.867866 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:50.868259 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.868776 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.869521 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:50.869733 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:50.869809 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:50.872512 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:34543: BlockingRecv error: recv got EOF from 127.16.61.3:34543 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 11:05:51 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:51 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:51.541370 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:51.536106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:45465)
0619 11:05:51.536391 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:51.536405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:51.536493 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:51.536735 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:51.536737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:51.536909 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:51.537000 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:51.537005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.537761 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.537764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:51.538322 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:51.538329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.538729 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.538732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:51.538791 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:51.539344 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:51.539356 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:51.541194 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":70,"threads_started":1}
W20260619 11:05:51.541464 25853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:51.536149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:45465 (local address 127.16.61.62:44377)
0619 11:05:51.536437 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:51.536440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:51.536453 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:05:51.536591 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:51.536593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:51.536632 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:51.536700 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:51.537125 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.537664 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.538420 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.538638 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.538712 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:51.541329 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:45465: BlockingRecv error: recv got EOF from 127.16.61.1:45465 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":70,"threads_started":1}
Jun 19 11:05:51 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:51 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:51.705521 25854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:51.700271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:50261)
0619 11:05:51.700519 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:51.700532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:51.700625 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:51.700850 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:51.700852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:51.701030 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:51.701117 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:51.701122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.701918 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.701921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:51.702530 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:51.702540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.702999 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.703002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:51.703062 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:51.703535 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:51.703546 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:51.705342 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
W20260619 11:05:51.705761 25853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:51.700319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:50261 (local address 127.16.61.62:44377)
0619 11:05:51.700449 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:51.700452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:51.700596 (+ 144us) server_negotiation.cc:407] Connection header received
0619 11:05:51.700700 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:51.700702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:51.700744 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:51.700814 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:51.701223 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.701740 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.702652 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.702869 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.703021 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:51.705510 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:50261: BlockingRecv error: recv got EOF from 127.16.61.2:50261 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 11:05:51 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:51 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.3@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:51.878492 25855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:51.873202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.3:55425)
0619 11:05:51.873471 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:51.873484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:51.873579 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:51.873829 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:51.873831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:51.874018 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:51.874111 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:51.874116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.874824 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.874826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:51.875393 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:51.875399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.875807 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.875810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:51.875867 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:51.876445 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:51.876458 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:51.878373 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
W20260619 11:05:51.878659 25853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:51.873262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:55425 (local address 127.16.61.62:44377)
0619 11:05:51.873399 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:51.873404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:51.873551 (+ 147us) server_negotiation.cc:407] Connection header received
0619 11:05:51.873665 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:51.873668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:51.873727 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:51.873796 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:51.874205 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.874738 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.875484 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:51.875720 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:51.875794 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:51.878495 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.3:55425: BlockingRecv error: recv got EOF from 127.16.61.3:55425 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 11:05:52 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:52 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.1@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:52.550765 25856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:52.542291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.1:52947)
0619 11:05:52.542550 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:52.542563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:52.542643 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:52.542887 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:52.542889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:52.543072 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:52.543158 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:52.543164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.543970 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.543972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:52.544521 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:52.544527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.544957 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.544959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:52.545016 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:52.548819 (+ 3803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:52.548830 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:52.550647 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
W20260619 11:05:52.550976 25857 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:52.542394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:52947 (local address 127.16.61.62:44377)
0619 11:05:52.542636 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:52.542640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:52.542654 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:05:52.542712 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:52.542714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:52.542766 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:52.542862 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:52.543264 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.543874 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.544616 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.544857 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.544937 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:52.550790 (+ 5853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.1:52947: BlockingRecv error: recv got EOF from 127.16.61.1:52947 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":58,"threads_started":1}
Jun 19 11:05:52 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 11:05:52 dist-test-slave-8vcz krb5kdc[24798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.61.2@KRBTEST.COM for oryx/127.16.61.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 11:05:52.711959 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:52.706433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:44377 (local address 127.16.61.2:57699)
0619 11:05:52.706718 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:52.706730 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:52.706850 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:52.707141 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:52.707144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:52.707361 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:52.707447 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:52.707453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.708284 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.708289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:52.708853 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:52.708861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.709342 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.709346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:52.709433 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:52.709974 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:52.709988 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:52.711837 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.61.62:44377: Server oryx/127.16.61.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":79,"threads_started":1}
W20260619 11:05:52.712106 25857 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:52.706488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:57699 (local address 127.16.61.62:44377)
0619 11:05:52.706627 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:52.706631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:52.706805 (+ 174us) server_negotiation.cc:407] Connection header received
0619 11:05:52.706973 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:52.706976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:52.707035 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:52.707117 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:52.707568 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.708173 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.708973 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.709221 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.709328 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:52.711958 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.61.2:57699: BlockingRecv error: recv got EOF from 127.16.61.2:57699 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:05:52.784086 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 25381
I20260619 11:05:52.789582 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 25533
I20260619 11:05:52.794788 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 25665
I20260619 11:05:52.799885 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 25311
2026-06-19T11:05:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17802 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T11:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:52Z Disabled control of system clock
I20260619 11:05:52.829087 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:34481
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:34481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 11:05:52.932677 25862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:52.932924 25862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:52.933023 25862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:52.936937 25862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:52.936988 25862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:52.937003 25862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:52.937021 25862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:52.937036 25862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:52.941637 25862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:34481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:34481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:52.942610 25862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:52.943903 25862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:05:52.952340 25862 server_base.cc:1034] running on GCE node
W20260619 11:05:52.952283 25870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:52.952247 25868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:52.952283 25867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:52.953058 25862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:52.954128 25862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:52.955335 25862 hybrid_clock.cc:648] HybridClock initialized: now 1781867152955323 us; error 23 us; skew 500 ppm
I20260619 11:05:52.957038 25862 webserver.cc:460] Webserver started at http://127.16.61.62:43007/ using document root <none> and password file <none>
I20260619 11:05:52.957582 25862 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:52.957671 25862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:52.957888 25862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:52.959833 25862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "901bae3743c2432c848e5e4f18ddfcc6"
format_stamp: "Formatted at 2026-06-19 11:05:52 on dist-test-slave-8vcz"
server_key: "3a087bf96f89adc7f16c9d6d1b2c7356"
server_key_iv: "4f449a11709680157c1c29dd0aa5a19f"
server_key_version: "encryptionkey@0"
I20260619 11:05:52.960378 25862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "901bae3743c2432c848e5e4f18ddfcc6"
format_stamp: "Formatted at 2026-06-19 11:05:52 on dist-test-slave-8vcz"
server_key: "3a087bf96f89adc7f16c9d6d1b2c7356"
server_key_iv: "4f449a11709680157c1c29dd0aa5a19f"
server_key_version: "encryptionkey@0"
I20260619 11:05:52.963817 25862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:05:52.966054 25876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:52.967152 25862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:52.967296 25862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "901bae3743c2432c848e5e4f18ddfcc6"
format_stamp: "Formatted at 2026-06-19 11:05:52 on dist-test-slave-8vcz"
server_key: "3a087bf96f89adc7f16c9d6d1b2c7356"
server_key_iv: "4f449a11709680157c1c29dd0aa5a19f"
server_key_version: "encryptionkey@0"
I20260619 11:05:52.967404 25862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:52.984400 25862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:52.984992 25862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:52.985172 25862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:52.991467 25862 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:34481
I20260619 11:05:52.991492 25928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:34481 every 8 connection(s)
I20260619 11:05:52.992332 25862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:52.994020 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 25862
I20260619 11:05:52.994107 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:52.994392 16628 external_mini_cluster.cc:1428] Setting key 102251d345a387eddb46b7473106597c
I20260619 11:05:52.995494 25929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:53.000640 25929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6: Bootstrap starting.
I20260619 11:05:53.002950 25929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:53.003288 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:52.995690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.16.61.62:34481)
0619 11:05:52.996277 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:52.996291 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:52.996324 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:05:52.996930 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:52.996946 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:52.997305 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:52.997613 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:52.998017 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:52.998929 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:52.999875 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.000332 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.000498 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:53.001377 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:53.001386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:53.001402 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:53.001787 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:53.001834 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:53.001854 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:53.001962 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:53.002173 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 11:05:53.002413 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260619 11:05:53.003789 25929 log.cc:824] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:53.005653 25929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6: No bootstrap required, opened a new log
I20260619 11:05:53.008968 25929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901bae3743c2432c848e5e4f18ddfcc6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 34481 } }
I20260619 11:05:53.009162 25929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:53.009195 25929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901bae3743c2432c848e5e4f18ddfcc6, State: Initialized, Role: FOLLOWER
I20260619 11:05:53.009572 25929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901bae3743c2432c848e5e4f18ddfcc6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 34481 } }
I20260619 11:05:53.009684 25929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:53.009768 25929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:53.009864 25929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:53.010689 25929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901bae3743c2432c848e5e4f18ddfcc6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 34481 } }
I20260619 11:05:53.011049 25929 leader_election.cc:302] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 901bae3743c2432c848e5e4f18ddfcc6; no voters:
I20260619 11:05:53.011364 25929 leader_election.cc:288] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:53.011457 25934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:53.011737 25934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [term 1 LEADER]: Becoming Leader. State: Replica: 901bae3743c2432c848e5e4f18ddfcc6, State: Running, Role: LEADER
I20260619 11:05:53.012064 25934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901bae3743c2432c848e5e4f18ddfcc6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 34481 } }
I20260619 11:05:53.012524 25929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:53.013602 25935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "901bae3743c2432c848e5e4f18ddfcc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901bae3743c2432c848e5e4f18ddfcc6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 34481 } } }
I20260619 11:05:53.013726 25935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:53.013931 25936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 901bae3743c2432c848e5e4f18ddfcc6. Latest consensus state: current_term: 1 leader_uuid: "901bae3743c2432c848e5e4f18ddfcc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901bae3743c2432c848e5e4f18ddfcc6" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 34481 } } }
I20260619 11:05:53.014120 25943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:53.014191 25936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:53.017012 25943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:53.017305 25951 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:05:53.022071 25943 catalog_manager.cc:1347] Generated new cluster ID: 094b936880b54ff887da386928069853
I20260619 11:05:53.022141 25943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:53.030037 25943 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:53.030800 25943 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:53.035918 25943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 901bae3743c2432c848e5e4f18ddfcc6: Generated new TSK 0
I20260619 11:05:53.036469 25943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:05:53.045282 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:34481
--builtin_ntp_servers=127.16.61.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 11:05:53.150413 25953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:53.150631 25953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:53.150694 25953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:53.154649 25953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:53.154691 25953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:53.154742 25953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:53.159277 25953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:53.160246 25953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:53.161478 25953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:53.170433 25959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:53.170526 25958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:53.170536 25953 server_base.cc:1034] running on GCE node
W20260619 11:05:53.170444 25961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:53.171109 25953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:53.171617 25953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:53.172819 25953 hybrid_clock.cc:648] HybridClock initialized: now 1781867153172806 us; error 24 us; skew 500 ppm
I20260619 11:05:53.174425 25953 webserver.cc:460] Webserver started at http://127.16.61.1:43499/ using document root <none> and password file <none>
I20260619 11:05:53.174961 25953 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:53.175047 25953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:53.175261 25953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:53.177175 25953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "80fd5162ae5943edb530f496b54132c0"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "75f6ec657ab2ecc8418132b81e8fcbed"
server_key_iv: "37a3d460c88337ed0ffe508c5b2f8235"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.177709 25953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "80fd5162ae5943edb530f496b54132c0"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "75f6ec657ab2ecc8418132b81e8fcbed"
server_key_iv: "37a3d460c88337ed0ffe508c5b2f8235"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.181071 25953 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:05:53.183281 25967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.184384 25953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:05:53.184516 25953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "80fd5162ae5943edb530f496b54132c0"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "75f6ec657ab2ecc8418132b81e8fcbed"
server_key_iv: "37a3d460c88337ed0ffe508c5b2f8235"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.184619 25953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:53.201192 25953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:53.201849 25953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:53.202052 25953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:53.202641 25953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:53.203513 25953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:53.203562 25953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.203629 25953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:53.203719 25953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.212402 25953 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:35439
I20260619 11:05:53.212426 26080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:35439 every 8 connection(s)
I20260619 11:05:53.213332 25953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:53.221176 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 25953
I20260619 11:05:53.221290 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:53.221576 16628 external_mini_cluster.cc:1428] Setting key 5fdcc64f5098c6e26bab189234a5e1c7
I20260619 11:05:53.221949 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.215055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:51849 (local address 127.16.61.62:34481)
0619 11:05:53.215225 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:53.215230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:53.216247 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 11:05:53.217502 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:53.217504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:53.217548 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:53.217619 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:53.218420 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.218936 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.220068 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.220281 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.220359 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:53.221429 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:53.221433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:53.221436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:53.221448 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:53.221473 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:53.221481 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:53.221566 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:53.221747 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 11:05:53.221808 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:05:53.222837 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.215453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:34481 (local address 127.16.61.1:51849)
0619 11:05:53.216106 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:53.216144 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:53.217327 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:53.217769 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:53.217775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:53.217851 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:53.218304 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:53.218315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.219053 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.219055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:53.219964 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:53.219973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.220401 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.220403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:53.220532 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:53.221128 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:53.221130 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:53.221194 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:53.221272 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:53.221583 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:53.221591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:53.221596 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:05:53.221734 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 11:05:53.221986 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":104,"threads_started":1}
I20260619 11:05:53.223464 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:34481
--builtin_ntp_servers=127.16.61.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 11:05:53.224081 26081 heartbeater.cc:347] Connected to a master server at 127.16.61.62:34481
I20260619 11:05:53.224311 26081 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:53.224768 26081 heartbeater.cc:511] Master 127.16.61.62:34481 requested a full tablet report, sending...
I20260619 11:05:53.226100 25893 ts_manager.cc:194] Registered new tserver with Master: 80fd5162ae5943edb530f496b54132c0 (127.16.61.1:35439)
I20260619 11:05:53.227167 25893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.1:51849
W20260619 11:05:53.329763 26084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:53.330024 26084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:53.330089 26084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:53.334015 26084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:53.334082 26084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:53.334165 26084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:53.338680 26084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:53.339592 26084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:53.340808 26084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:53.349651 26092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:53.349711 26089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:53.349825 26084 server_base.cc:1034] running on GCE node
W20260619 11:05:53.349699 26090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:53.350286 26084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:53.350801 26084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:53.351946 26084 hybrid_clock.cc:648] HybridClock initialized: now 1781867153351933 us; error 24 us; skew 500 ppm
I20260619 11:05:53.353542 26084 webserver.cc:460] Webserver started at http://127.16.61.2:37751/ using document root <none> and password file <none>
I20260619 11:05:53.354015 26084 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:53.354070 26084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:53.354215 26084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:53.356056 26084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "d352ac7d28454255a45ad2b993e0de45"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "a10ad7153b93c1acbf2d18774fccc1b1"
server_key_iv: "adec68a73df7f9b5b71113baf2aa4e2c"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.356528 26084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "d352ac7d28454255a45ad2b993e0de45"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "a10ad7153b93c1acbf2d18774fccc1b1"
server_key_iv: "adec68a73df7f9b5b71113baf2aa4e2c"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.359861 26084 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:53.361979 26098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.363051 26084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:53.363142 26084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "d352ac7d28454255a45ad2b993e0de45"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "a10ad7153b93c1acbf2d18774fccc1b1"
server_key_iv: "adec68a73df7f9b5b71113baf2aa4e2c"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.363219 26084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:53.387631 26084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:53.388300 26084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:53.388432 26084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:53.388890 26084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:53.389736 26084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:53.389780 26084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.389816 26084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:53.389839 26084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.397900 26084 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:37565
I20260619 11:05:53.397938 26211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:37565 every 8 connection(s)
I20260619 11:05:53.398698 26084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:53.399217 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 26084
I20260619 11:05:53.399319 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:53.399653 16628 external_mini_cluster.cc:1428] Setting key 8b20fd3f11b9eb869507325d65e6eb9b
I20260619 11:05:53.401705 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:34481
--builtin_ntp_servers=127.16.61.20:45211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 11:05:53.409065 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.400489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:37867 (local address 127.16.61.62:34481)
0619 11:05:53.400626 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:53.400630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:53.403058 (+ 2428us) server_negotiation.cc:407] Connection header received
0619 11:05:53.404464 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:53.404467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:53.404527 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:53.404611 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:53.405516 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.406025 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.407127 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.407357 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.407467 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:53.408583 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:53.408586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:53.408589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:53.408604 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:53.408627 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:53.408635 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:53.408709 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:53.408870 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 11:05:53.408928 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:05:53.409837 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.400935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:34481 (local address 127.16.61.2:37867)
0619 11:05:53.402889 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:53.402950 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:53.404274 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:53.404780 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:53.404786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:53.404903 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:53.405373 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:53.405384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.406145 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.406149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:53.406993 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:53.407002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.407500 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.407504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:53.407733 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:53.408324 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:53.408325 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:53.408362 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:53.408420 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:53.408722 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:53.408734 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:53.408744 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 11:05:53.408857 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 11:05:53.409129 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476,"thread_start_us":129,"threads_started":1}
I20260619 11:05:53.410884 26212 heartbeater.cc:347] Connected to a master server at 127.16.61.62:34481
I20260619 11:05:53.411124 26212 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:53.411597 26212 heartbeater.cc:511] Master 127.16.61.62:34481 requested a full tablet report, sending...
I20260619 11:05:53.412621 25893 ts_manager.cc:194] Registered new tserver with Master: d352ac7d28454255a45ad2b993e0de45 (127.16.61.2:37565)
I20260619 11:05:53.413159 25893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.2:37867
W20260619 11:05:53.508070 26215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:53.508325 26215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:53.508394 26215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:53.512279 26215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:53.512348 26215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:53.512449 26215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:53.517158 26215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:45211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:34481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:53.518079 26215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:53.519315 26215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:53.527972 26223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:53.527972 26221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:53.528010 26220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:53.528273 26215 server_base.cc:1034] running on GCE node
I20260619 11:05:53.528661 26215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:53.529213 26215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:53.530390 26215 hybrid_clock.cc:648] HybridClock initialized: now 1781867153530351 us; error 47 us; skew 500 ppm
I20260619 11:05:53.532189 26215 webserver.cc:460] Webserver started at http://127.16.61.3:45565/ using document root <none> and password file <none>
I20260619 11:05:53.532689 26215 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:53.532747 26215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:53.532893 26215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:53.534698 26215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "12f0187f95c9419f88ac5922d55a9b60"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "2998a122f8292aff902e827ce8446e01"
server_key_iv: "aa210557807a8a5bd93ecfdf47b61ef7"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.535171 26215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "12f0187f95c9419f88ac5922d55a9b60"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "2998a122f8292aff902e827ce8446e01"
server_key_iv: "aa210557807a8a5bd93ecfdf47b61ef7"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.538810 26215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:05:53.541091 26229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.542230 26215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:53.542317 26215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "12f0187f95c9419f88ac5922d55a9b60"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "2998a122f8292aff902e827ce8446e01"
server_key_iv: "aa210557807a8a5bd93ecfdf47b61ef7"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.542398 26215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:53.565531 26215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:53.566241 26215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:53.566388 26215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:53.566967 26215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:53.567957 26215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:53.568005 26215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.568045 26215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:53.568063 26215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.576325 26215 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:39029
I20260619 11:05:53.576349 26342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:39029 every 8 connection(s)
I20260619 11:05:53.577150 26215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:53.577497 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 26215
I20260619 11:05:53.577615 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:53.577921 16628 external_mini_cluster.cc:1428] Setting key 03b28b08d20300d5ba04a856c26e442b
I20260619 11:05:53.585518 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.578839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:46133 (local address 127.16.61.62:34481)
0619 11:05:53.578942 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:53.578952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:53.580176 (+ 1224us) server_negotiation.cc:407] Connection header received
0619 11:05:53.581456 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:53.581458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:53.581499 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:53.581573 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:53.582374 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.582881 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.583935 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.584140 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.584218 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:53.585140 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:53.585143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:53.585145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:53.585156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:53.585174 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:53.585181 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:53.585237 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:53.585363 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 11:05:53.585408 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 11:05:53.586310 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.579327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:34481 (local address 127.16.61.3:46133)
0619 11:05:53.580031 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:53.580079 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:53.581289 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:53.581725 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:53.581731 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:53.581807 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:53.582264 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:53.582277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.582987 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.582989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:53.583836 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:53.583844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:53.584260 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:53.584263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:53.584396 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:53.584913 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:53.584914 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:53.584956 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:53.585013 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:53.585250 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:53.585255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:53.585259 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:05:53.585348 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 11:05:53.585544 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":144,"threads_started":1}
I20260619 11:05:53.587342 26343 heartbeater.cc:347] Connected to a master server at 127.16.61.62:34481
I20260619 11:05:53.587594 26343 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:53.588085 26343 heartbeater.cc:511] Master 127.16.61.62:34481 requested a full tablet report, sending...
I20260619 11:05:53.589038 25893 ts_manager.cc:194] Registered new tserver with Master: 12f0187f95c9419f88ac5922d55a9b60 (127.16.61.3:39029)
I20260619 11:05:53.589576 25893 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.3:46133
I20260619 11:05:53.591552 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:05:53.594218 25932 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:53.593296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.16.61.62:34481)
0619 11:05:53.593436 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:53.593440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:53.593616 (+ 176us) server_negotiation.cc:407] Connection header received
0619 11:05:53.593764 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:53.593766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:53.593805 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:53.593889 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:53.594114 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46686: BlockingRecv error: recv got EOF from 127.0.0.1:46686 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:05:53.594367 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 25953
I20260619 11:05:53.599419 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 26084
I20260619 11:05:53.604460 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 26215
I20260619 11:05:53.609515 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 25862
2026-06-19T11:05:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (808 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T11:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:53Z Disabled control of system clock
I20260619 11:05:53.637259 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:32835
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 11:05:53.740020 26354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:53.740236 26354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:53.740272 26354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:53.744071 26354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:53.744112 26354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:53.744127 26354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:53.744143 26354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:53.744158 26354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:53.748574 26354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:32835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:53.749423 26354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:53.750602 26354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:53.758383 26359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:53.758387 26360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:53.758500 26362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:53.758481 26354 server_base.cc:1034] running on GCE node
I20260619 11:05:53.759115 26354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:53.760125 26354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:53.761281 26354 hybrid_clock.cc:648] HybridClock initialized: now 1781867153761264 us; error 28 us; skew 500 ppm
I20260619 11:05:53.762990 26354 webserver.cc:460] Webserver started at http://127.16.61.62:46199/ using document root <none> and password file <none>
I20260619 11:05:53.763515 26354 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:53.763573 26354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:53.763836 26354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:53.765690 26354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "930001fe7ad84cceb15c7778cd775e63"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "46483b1e810b1350f966ad1e6e26e2b7"
server_key_iv: "43dd359112f87d3f459d2dcd19e37f48"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.766204 26354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "930001fe7ad84cceb15c7778cd775e63"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "46483b1e810b1350f966ad1e6e26e2b7"
server_key_iv: "43dd359112f87d3f459d2dcd19e37f48"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.769552 26354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:05:53.771616 26368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.772696 26354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:05:53.772825 26354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "930001fe7ad84cceb15c7778cd775e63"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "46483b1e810b1350f966ad1e6e26e2b7"
server_key_iv: "43dd359112f87d3f459d2dcd19e37f48"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.772931 26354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:53.777105 26354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:53.777637 26354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:53.777807 26354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:53.784302 26354 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:32835
I20260619 11:05:53.784327 26420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:32835 every 8 connection(s)
I20260619 11:05:53.785117 26354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:53.788082 26421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:53.792008 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 26354
I20260619 11:05:53.792140 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:53.792405 16628 external_mini_cluster.cc:1428] Setting key 6c621134ab21397ad34c8734440cc89d
I20260619 11:05:53.793407 26421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63: Bootstrap starting.
I20260619 11:05:53.795542 26421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:53.796439 26421 log.cc:824] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:53.797655 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.793735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.16.61.62:32835)
0619 11:05:53.794298 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:53.794312 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:53.794344 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:05:53.794933 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:53.794950 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:53.795267 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:53.795533 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:53.796370 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:53.796375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:53.796386 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:53.796395 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:53.796453 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:53.796469 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:53.796609 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:53.796717 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 11:05:53.796955 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":128,"threads_started":1}
I20260619 11:05:53.798305 26421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63: No bootstrap required, opened a new log
I20260619 11:05:53.801936 26421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930001fe7ad84cceb15c7778cd775e63" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32835 } }
I20260619 11:05:53.802132 26421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:53.802209 26421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930001fe7ad84cceb15c7778cd775e63, State: Initialized, Role: FOLLOWER
I20260619 11:05:53.802582 26421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930001fe7ad84cceb15c7778cd775e63" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32835 } }
I20260619 11:05:53.802718 26421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:53.802778 26421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:53.802899 26421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:53.803889 26421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930001fe7ad84cceb15c7778cd775e63" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32835 } }
I20260619 11:05:53.804237 26421 leader_election.cc:302] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 930001fe7ad84cceb15c7778cd775e63; no voters:
I20260619 11:05:53.804515 26421 leader_election.cc:288] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:53.804615 26426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:53.804832 26426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [term 1 LEADER]: Becoming Leader. State: Replica: 930001fe7ad84cceb15c7778cd775e63, State: Running, Role: LEADER
I20260619 11:05:53.805150 26426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930001fe7ad84cceb15c7778cd775e63" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32835 } }
I20260619 11:05:53.805747 26421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:53.806798 26428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 930001fe7ad84cceb15c7778cd775e63. Latest consensus state: current_term: 1 leader_uuid: "930001fe7ad84cceb15c7778cd775e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930001fe7ad84cceb15c7778cd775e63" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32835 } } }
I20260619 11:05:53.806826 26427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "930001fe7ad84cceb15c7778cd775e63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930001fe7ad84cceb15c7778cd775e63" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32835 } } }
I20260619 11:05:53.807018 26428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:53.807037 26427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:53.807753 26435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:53.810117 26435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:53.814781 26435 catalog_manager.cc:1347] Generated new cluster ID: 094bc9b7d905417ebfd18d85746c3c36
I20260619 11:05:53.814841 26435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:53.822345 26435 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:53.822640 26435 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:53.826937 26435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 930001fe7ad84cceb15c7778cd775e63: Generated new TSK 0
I20260619 11:05:53.827490 26435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:05:53.835515 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32835
--builtin_ntp_servers=127.16.61.20:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 11:05:53.940666 26445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:53.940892 26445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:53.940930 26445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:53.944850 26445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:53.944891 26445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:53.944954 26445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:53.949483 26445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:53.950454 26445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:53.951713 26445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:53.960451 26451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:53.960531 26453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:53.960525 26445 server_base.cc:1034] running on GCE node
W20260619 11:05:53.960670 26450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:53.960992 26445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:53.961555 26445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:53.962702 26445 hybrid_clock.cc:648] HybridClock initialized: now 1781867153962689 us; error 25 us; skew 500 ppm
I20260619 11:05:53.964411 26445 webserver.cc:460] Webserver started at http://127.16.61.1:45453/ using document root <none> and password file <none>
I20260619 11:05:53.964942 26445 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:53.965001 26445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:53.965240 26445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:53.967096 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "b4b93a95dd9345028bf08cf9d0edac2c"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "6b62fc766b217bd7fc64b23f2525b431"
server_key_iv: "5765e01082a698986943884e6f5fd94c"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.967654 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "b4b93a95dd9345028bf08cf9d0edac2c"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "6b62fc766b217bd7fc64b23f2525b431"
server_key_iv: "5765e01082a698986943884e6f5fd94c"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.971063 26445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:05:53.973332 26459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.974386 26445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 11:05:53.974498 26445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "b4b93a95dd9345028bf08cf9d0edac2c"
format_stamp: "Formatted at 2026-06-19 11:05:53 on dist-test-slave-8vcz"
server_key: "6b62fc766b217bd7fc64b23f2525b431"
server_key_iv: "5765e01082a698986943884e6f5fd94c"
server_key_version: "encryptionkey@0"
I20260619 11:05:53.974608 26445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:53.979137 26445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:53.979746 26445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:53.979915 26445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:53.980427 26445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:53.981263 26445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:53.981333 26445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.981395 26445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:53.981438 26445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:53.989372 26445 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:34213
I20260619 11:05:53.989400 26572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:34213 every 8 connection(s)
I20260619 11:05:53.990166 26445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:05:53.990797 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 26445
I20260619 11:05:53.990914 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:53.991209 16628 external_mini_cluster.cc:1428] Setting key 4148d65c410b51fdd64e98150f0f9e1b
I20260619 11:05:53.993306 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32835
--builtin_ntp_servers=127.16.61.20:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 11:05:53.996188 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.991954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:59631 (local address 127.16.61.62:32835)
0619 11:05:53.992082 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:53.992091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:53.993209 (+ 1118us) server_negotiation.cc:407] Connection header received
0619 11:05:53.994508 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:53.994511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:53.994549 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:53.994622 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:53.995722 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:53.995726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:53.995728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:53.995740 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:53.995761 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:53.995770 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:53.995876 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:53.996022 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 11:05:53.996063 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:53.997025 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:53.992452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32835 (local address 127.16.61.1:59631)
0619 11:05:53.993080 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:53.993116 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:53.994309 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:53.994772 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:53.994778 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:53.994857 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:53.995443 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:53.995445 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:53.995483 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:53.995546 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:53.995862 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:53.995870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:53.995875 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 11:05:53.996061 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 11:05:53.996278 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":98,"threads_started":1}
I20260619 11:05:53.998294 26573 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32835
I20260619 11:05:53.998512 26573 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:53.998873 26573 heartbeater.cc:511] Master 127.16.61.62:32835 requested a full tablet report, sending...
I20260619 11:05:54.000473 26385 ts_manager.cc:194] Registered new tserver with Master: b4b93a95dd9345028bf08cf9d0edac2c (127.16.61.1:34213)
W20260619 11:05:54.099262 26576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:54.099489 26576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:54.099534 26576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:54.103389 26576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:54.103437 26576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:54.103490 26576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:54.108362 26576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:54.109233 26576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:54.110428 26576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:54.119541 26584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:54.119578 26582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:54.119702 26576 server_base.cc:1034] running on GCE node
W20260619 11:05:54.119540 26581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:54.120266 26576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:54.120905 26576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:54.122103 26576 hybrid_clock.cc:648] HybridClock initialized: now 1781867154122086 us; error 28 us; skew 500 ppm
I20260619 11:05:54.123961 26576 webserver.cc:460] Webserver started at http://127.16.61.2:43725/ using document root <none> and password file <none>
I20260619 11:05:54.124497 26576 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:54.124586 26576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:54.124797 26576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:54.126673 26576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "77351f1844e848d18882b85a68556119"
format_stamp: "Formatted at 2026-06-19 11:05:54 on dist-test-slave-8vcz"
server_key: "067a3bd06f9d13dea90a30210223a08b"
server_key_iv: "e8d7fdc540fe7237f074a666218ea90f"
server_key_version: "encryptionkey@0"
I20260619 11:05:54.127216 26576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "77351f1844e848d18882b85a68556119"
format_stamp: "Formatted at 2026-06-19 11:05:54 on dist-test-slave-8vcz"
server_key: "067a3bd06f9d13dea90a30210223a08b"
server_key_iv: "e8d7fdc540fe7237f074a666218ea90f"
server_key_version: "encryptionkey@0"
I20260619 11:05:54.130710 26576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:05:54.133062 26590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:54.134232 26576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:54.134455 26576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "77351f1844e848d18882b85a68556119"
format_stamp: "Formatted at 2026-06-19 11:05:54 on dist-test-slave-8vcz"
server_key: "067a3bd06f9d13dea90a30210223a08b"
server_key_iv: "e8d7fdc540fe7237f074a666218ea90f"
server_key_version: "encryptionkey@0"
I20260619 11:05:54.134564 26576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:54.138788 26576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:54.139339 26576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:54.139509 26576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:54.140062 26576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:54.140910 26576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:54.140980 26576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:54.141045 26576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:54.141098 26576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:54.149833 26576 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:44083
I20260619 11:05:54.149860 26703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:44083 every 8 connection(s)
I20260619 11:05:54.150638 26576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:54.156677 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:54.152673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:34859 (local address 127.16.61.62:32835)
0619 11:05:54.152805 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:54.152809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:54.153791 (+ 982us) server_negotiation.cc:407] Connection header received
0619 11:05:54.155030 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:54.155033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:54.155075 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:54.155163 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:54.156200 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:54.156203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:54.156206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:54.156217 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:54.156237 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:54.156246 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:54.156349 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:54.156505 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 11:05:54.156546 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:05:54.157402 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:54.153038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32835 (local address 127.16.61.2:34859)
0619 11:05:54.153647 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:54.153684 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:54.154831 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:54.155322 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:54.155328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:54.155409 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:54.155964 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:54.155965 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:54.156000 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:54.156061 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:54.156336 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:54.156343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:54.156349 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 11:05:54.156545 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 11:05:54.156761 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260619 11:05:54.158394 26704 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32835
I20260619 11:05:54.158577 26704 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:54.158906 26704 heartbeater.cc:511] Master 127.16.61.62:32835 requested a full tablet report, sending...
I20260619 11:05:54.159067 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 26576
I20260619 11:05:54.159191 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:54.159471 16628 external_mini_cluster.cc:1428] Setting key 2c5011fa45b739f483201a0b28098aa1
I20260619 11:05:54.159967 26385 ts_manager.cc:194] Registered new tserver with Master: 77351f1844e848d18882b85a68556119 (127.16.61.2:44083)
I20260619 11:05:54.161477 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32835
--builtin_ntp_servers=127.16.61.20:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 11:05:54.266762 26707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:54.267023 26707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:54.267098 26707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:54.271166 26707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:54.271229 26707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:54.271327 26707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:54.275921 26707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:54.276835 26707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:54.278168 26707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:54.287313 26713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:54.287354 26707 server_base.cc:1034] running on GCE node
W20260619 11:05:54.287320 26712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:54.287319 26715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:54.287935 26707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:54.288542 26707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:54.289697 26707 hybrid_clock.cc:648] HybridClock initialized: now 1781867154289683 us; error 25 us; skew 500 ppm
I20260619 11:05:54.291447 26707 webserver.cc:460] Webserver started at http://127.16.61.3:37627/ using document root <none> and password file <none>
I20260619 11:05:54.292034 26707 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:54.292095 26707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:54.292336 26707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:54.294224 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "04fa2f08c6684dbb951c5cb7e4001587"
format_stamp: "Formatted at 2026-06-19 11:05:54 on dist-test-slave-8vcz"
server_key: "2e1d41d4f4230900030f9f8c8a7326be"
server_key_iv: "883f9bfe892e6be726bdf0deef00da82"
server_key_version: "encryptionkey@0"
I20260619 11:05:54.294756 26707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "04fa2f08c6684dbb951c5cb7e4001587"
format_stamp: "Formatted at 2026-06-19 11:05:54 on dist-test-slave-8vcz"
server_key: "2e1d41d4f4230900030f9f8c8a7326be"
server_key_iv: "883f9bfe892e6be726bdf0deef00da82"
server_key_version: "encryptionkey@0"
I20260619 11:05:54.298283 26707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:05:54.300675 26721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:54.301755 26707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:54.301896 26707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "04fa2f08c6684dbb951c5cb7e4001587"
format_stamp: "Formatted at 2026-06-19 11:05:54 on dist-test-slave-8vcz"
server_key: "2e1d41d4f4230900030f9f8c8a7326be"
server_key_iv: "883f9bfe892e6be726bdf0deef00da82"
server_key_version: "encryptionkey@0"
I20260619 11:05:54.302016 26707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:54.306453 26707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:54.307055 26707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:54.307221 26707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:54.307771 26707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:54.308614 26707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:54.308681 26707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:54.308746 26707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:54.308799 26707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:54.317184 26707 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:45167
I20260619 11:05:54.317214 26834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:45167 every 8 connection(s)
I20260619 11:05:54.317951 26707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:54.323295 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:54.319559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:34325 (local address 127.16.61.62:32835)
0619 11:05:54.319741 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:54.319745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:54.320673 (+ 928us) server_negotiation.cc:407] Connection header received
0619 11:05:54.321834 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:54.321837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:54.321871 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:54.321941 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:54.322968 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:54.322971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:54.322973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:05:54.322983 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:54.323000 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:05:54.323007 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:54.323076 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:54.323166 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 11:05:54.323199 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:05:54.324118 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:54.319969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32835 (local address 127.16.61.3:34325)
0619 11:05:54.320541 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:54.320577 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:54.321661 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:54.322087 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:54.322093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:54.322172 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:54.322734 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:05:54.322735 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:54.322771 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:05:54.322832 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:54.323064 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:54.323068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:54.323071 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:05:54.323183 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 11:05:54.323378 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
I20260619 11:05:54.325191 26835 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32835
I20260619 11:05:54.325371 26835 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:54.325681 26835 heartbeater.cc:511] Master 127.16.61.62:32835 requested a full tablet report, sending...
I20260619 11:05:54.326647 26384 ts_manager.cc:194] Registered new tserver with Master: 04fa2f08c6684dbb951c5cb7e4001587 (127.16.61.3:45167)
I20260619 11:05:54.326678 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 26707
I20260619 11:05:54.326787 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:54.327121 16628 external_mini_cluster.cc:1428] Setting key 04376bfede09232a2925b5a6a0590c94
I20260619 11:05:54.329119 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 11:05:54.331772 26424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:05:54.330850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.16.61.62:32835)
0619 11:05:54.330988 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:54.330993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:54.331162 (+ 169us) server_negotiation.cc:407] Connection header received
0619 11:05:54.331293 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:54.331296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:54.331325 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:54.331412 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:54.331644 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51372: BlockingRecv error: recv got EOF from 127.0.0.1:51372 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:05:54.331911 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 26445
I20260619 11:05:54.337811 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 26576
I20260619 11:05:54.343268 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 26707
I20260619 11:05:54.348562 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 26354
2026-06-19T11:05:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (739 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:54 dist-test-slave-8vcz krb5kdc[26845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:54 dist-test-slave-8vcz krb5kdc[26845](info): set up 2 sockets
Jun 19 11:05:54 dist-test-slave-8vcz krb5kdc[26845](info): commencing operation
krb5kdc: starting...
W20260619 11:05:55.384657 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:55 dist-test-slave-8vcz krb5kdc[26845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:05:55.498431 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:46877
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:46877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:55.602113 26858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:55.602349 26858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:55.602387 26858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:55.606321 26858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:05:55.606370 26858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:55.606388 26858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:55.606458 26858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:05:55.606494 26858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:05:55.611575 26858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:46877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:46877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:55.612496 26858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:55.613415 26858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:55.621348 26866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:55.621477 26864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:55.621359 26863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:55.621413 26858 server_base.cc:1034] running on GCE node
I20260619 11:05:55.622107 26858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:55.623057 26858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:55.624207 26858 hybrid_clock.cc:648] HybridClock initialized: now 1781867155624179 us; error 40 us; skew 500 ppm
Jun 19 11:05:55 dist-test-slave-8vcz krb5kdc[26845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867155, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:55.626813 26858 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:05:55.627729 26858 webserver.cc:460] Webserver started at http://127.16.61.62:34515/ using document root <none> and password file <none>
I20260619 11:05:55.628253 26858 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:55.628341 26858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:55.628558 26858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:55.630439 26858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "c638916fc54a4cd3a674ce83edbb0442"
format_stamp: "Formatted at 2026-06-19 11:05:55 on dist-test-slave-8vcz"
server_key: "18faea8b8da8024d5ad9f5562df9d8cf"
server_key_iv: "f14a81cea77b784d4d27eae92901c65e"
server_key_version: "encryptionkey@0"
I20260619 11:05:55.630978 26858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "c638916fc54a4cd3a674ce83edbb0442"
format_stamp: "Formatted at 2026-06-19 11:05:55 on dist-test-slave-8vcz"
server_key: "18faea8b8da8024d5ad9f5562df9d8cf"
server_key_iv: "f14a81cea77b784d4d27eae92901c65e"
server_key_version: "encryptionkey@0"
I20260619 11:05:55.634474 26858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:05:55.636682 26873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:55.637820 26858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:05:55.637951 26858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "c638916fc54a4cd3a674ce83edbb0442"
format_stamp: "Formatted at 2026-06-19 11:05:55 on dist-test-slave-8vcz"
server_key: "18faea8b8da8024d5ad9f5562df9d8cf"
server_key_iv: "f14a81cea77b784d4d27eae92901c65e"
server_key_version: "encryptionkey@0"
I20260619 11:05:55.638067 26858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:55.658839 26858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:55.661993 26858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:55.662199 26858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:55.668915 26858 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:46877
I20260619 11:05:55.668929 26925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:46877 every 8 connection(s)
I20260619 11:05:55.669793 26858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:05:55.672457 26926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:55.673616 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 26858
I20260619 11:05:55.673735 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:05:55.674037 16628 external_mini_cluster.cc:1428] Setting key 32d0c0a1a782286770f3df7c07d3f2e5
I20260619 11:05:55.678610 26926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442: Bootstrap starting.
I20260619 11:05:55.681181 26926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:55.682075 26926 log.cc:824] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:05:55 dist-test-slave-8vcz krb5kdc[26845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:55.684165 26926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442: No bootstrap required, opened a new log
I20260619 11:05:55.688235 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:55.675294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.16.61.62:46877)
0619 11:05:55.675653 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:55.675700 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:55.675733 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:05:55.676340 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:55.676355 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:55.676660 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:55.676925 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:55.677851 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:55.678830 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:55.680000 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:55.680440 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:55.680553 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:55.683309 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:55.683318 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:55.683336 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:55.683366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:55.685256 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:55.685647 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:55.685654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:55.685660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:55.685787 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:55.686091 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:55.686097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:55.686099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:55.686647 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:55.686827 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:55.687020 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 11:05:55.687303 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
I20260619 11:05:55.688627 26926 raft_consensus.cc:348] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c638916fc54a4cd3a674ce83edbb0442" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46877 } }
I20260619 11:05:55.688851 26926 raft_consensus.cc:374] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:55.688910 26926 raft_consensus.cc:729] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c638916fc54a4cd3a674ce83edbb0442, State: Initialized, Role: FOLLOWER
I20260619 11:05:55.689378 26926 consensus_queue.cc:260] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c638916fc54a4cd3a674ce83edbb0442" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46877 } }
I20260619 11:05:55.689533 26926 raft_consensus.cc:388] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:05:55.689594 26926 raft_consensus.cc:482] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:05:55.689687 26926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:55.690632 26926 raft_consensus.cc:504] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c638916fc54a4cd3a674ce83edbb0442" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46877 } }
I20260619 11:05:55.690896 26926 leader_election.cc:302] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c638916fc54a4cd3a674ce83edbb0442; no voters:
I20260619 11:05:55.691123 26926 leader_election.cc:288] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:05:55.691236 26931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:55.691485 26931 raft_consensus.cc:686] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [term 1 LEADER]: Becoming Leader. State: Replica: c638916fc54a4cd3a674ce83edbb0442, State: Running, Role: LEADER
I20260619 11:05:55.691828 26931 consensus_queue.cc:237] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c638916fc54a4cd3a674ce83edbb0442" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46877 } }
I20260619 11:05:55.692247 26926 sys_catalog.cc:564] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:05:55.693454 26932 sys_catalog.cc:455] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c638916fc54a4cd3a674ce83edbb0442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c638916fc54a4cd3a674ce83edbb0442" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46877 } } }
I20260619 11:05:55.693527 26933 sys_catalog.cc:455] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c638916fc54a4cd3a674ce83edbb0442. Latest consensus state: current_term: 1 leader_uuid: "c638916fc54a4cd3a674ce83edbb0442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c638916fc54a4cd3a674ce83edbb0442" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46877 } } }
I20260619 11:05:55.693610 26932 sys_catalog.cc:458] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:55.693639 26933 sys_catalog.cc:458] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442 [sys.catalog]: This master's current role is: LEADER
I20260619 11:05:55.693977 26940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:05:55.696938 26940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:05:55.702278 26940 catalog_manager.cc:1347] Generated new cluster ID: 84fdb800891a45f9b305fac848c0483b
I20260619 11:05:55.702337 26940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:05:55.734429 26940 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:05:55.735713 26940 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:05:55.746816 26940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442: Generated new TSK 0
I20260619 11:05:55.747464 26940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:05:55.784355 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46877
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:55.889099 26952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:55.889355 26952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:55.889428 26952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:55.893533 26952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:55.893618 26952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:55.893711 26952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:05:55.898710 26952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:55.899646 26952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:55.900609 26952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:55.909539 26960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:55.909559 26957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:55.909560 26958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:55.909745 26952 server_base.cc:1034] running on GCE node
I20260619 11:05:55.910229 26952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:55.910821 26952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:55.911998 26952 hybrid_clock.cc:648] HybridClock initialized: now 1781867155911966 us; error 43 us; skew 500 ppm
Jun 19 11:05:55 dist-test-slave-8vcz krb5kdc[26845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867155, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:55.914657 26952 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:05:55.915436 26952 webserver.cc:460] Webserver started at http://127.16.61.1:34651/ using document root <none> and password file <none>
I20260619 11:05:55.915992 26952 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:55.916054 26952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:55.916205 26952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:55.918035 26952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "1a2dec33e86c4711ab48d1c229616043"
format_stamp: "Formatted at 2026-06-19 11:05:55 on dist-test-slave-8vcz"
server_key: "6cfe41fe2310943f3668c72094313c0a"
server_key_iv: "9c764e1e12bf5b5041b801ae154248d1"
server_key_version: "encryptionkey@0"
I20260619 11:05:55.918527 26952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "1a2dec33e86c4711ab48d1c229616043"
format_stamp: "Formatted at 2026-06-19 11:05:55 on dist-test-slave-8vcz"
server_key: "6cfe41fe2310943f3668c72094313c0a"
server_key_iv: "9c764e1e12bf5b5041b801ae154248d1"
server_key_version: "encryptionkey@0"
I20260619 11:05:55.921981 26952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:05:55.924157 26967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:55.925272 26952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:55.925364 26952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "1a2dec33e86c4711ab48d1c229616043"
format_stamp: "Formatted at 2026-06-19 11:05:55 on dist-test-slave-8vcz"
server_key: "6cfe41fe2310943f3668c72094313c0a"
server_key_iv: "9c764e1e12bf5b5041b801ae154248d1"
server_key_version: "encryptionkey@0"
I20260619 11:05:55.925443 26952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:55.936396 26952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:55.939584 26952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:55.939805 26952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:55.940306 26952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:55.941160 26952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:55.941237 26952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:55.941274 26952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:55.941295 26952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:55.949831 26952 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:41433
I20260619 11:05:55.949854 27080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:41433 every 8 connection(s)
I20260619 11:05:55.950704 26952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:05:55 dist-test-slave-8vcz krb5kdc[26845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867155, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:55.959982 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 26952
I20260619 11:05:55.960117 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:05:55.960388 16628 external_mini_cluster.cc:1428] Setting key 46d46bd4093abe151c42ed0abe1b1620
I20260619 11:05:55.964178 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:55.952456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:60151 (local address 127.16.61.62:46877)
0619 11:05:55.952582 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:55.952586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:55.953363 (+ 777us) server_negotiation.cc:407] Connection header received
0619 11:05:55.954724 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:55.954727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:55.954783 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:55.954886 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:55.956050 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:55.956583 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:55.957695 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:55.957939 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:55.958055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:55.960485 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:55.960490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:55.960492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:55.960518 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:55.962137 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:55.962599 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:55.962606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:55.962610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:55.962702 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:55.962999 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:55.963005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:55.963010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:55.963243 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:55.963347 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:55.963829 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 11:05:55.963969 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:05:55.964841 27083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:55.952825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46877 (local address 127.16.61.1:60151)
0619 11:05:55.953228 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:55.953267 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:55.954514 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:55.955053 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:55.955060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:55.955393 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:55.955907 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:55.955921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:55.956706 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:55.956709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:55.957561 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:55.957569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:55.958086 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:55.958090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:55.958255 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:55.958830 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:55.958844 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:55.960322 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:55.962242 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:55.962247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:55.962257 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:55.962490 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:55.962792 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:55.962794 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:55.962796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:55.962912 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:55.963340 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:55.963345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:55.963556 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 11:05:55.963772 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 11:05:55.964005 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20260619 11:05:55.965969 27081 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46877
I20260619 11:05:55.966207 27081 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:55.966747 27081 heartbeater.cc:511] Master 127.16.61.62:46877 requested a full tablet report, sending...
I20260619 11:05:55.968401 26890 ts_manager.cc:194] Registered new tserver with Master: 1a2dec33e86c4711ab48d1c229616043 (127.16.61.1:41433)
I20260619 11:05:55.969986 26890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:60151
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:05:55.986474 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46877
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:56.092722 27086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:56.093046 27086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:56.093107 27086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:56.097105 27086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:56.097155 27086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:56.097219 27086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:05:56.102277 27086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:56.103199 27086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:56.104130 27086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:05:56.113421 27091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:56.113408 27092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:56.113440 27094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:56.113644 27086 server_base.cc:1034] running on GCE node
I20260619 11:05:56.114143 27086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:56.114748 27086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:56.115936 27086 hybrid_clock.cc:648] HybridClock initialized: now 1781867156115920 us; error 31 us; skew 500 ppm
Jun 19 11:05:56 dist-test-slave-8vcz krb5kdc[26845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867156, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:56.118556 27086 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:05:56.119392 27086 webserver.cc:460] Webserver started at http://127.16.61.2:43425/ using document root <none> and password file <none>
I20260619 11:05:56.119992 27086 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:56.120081 27086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:56.120308 27086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:56.122257 27086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "8fa2cd2670944b70a3569598c5031a5e"
format_stamp: "Formatted at 2026-06-19 11:05:56 on dist-test-slave-8vcz"
server_key: "59493604a8b65a61da287b21cc7d2da6"
server_key_iv: "75afb687929600f1f2b5b9bb5d121ac4"
server_key_version: "encryptionkey@0"
I20260619 11:05:56.122804 27086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "8fa2cd2670944b70a3569598c5031a5e"
format_stamp: "Formatted at 2026-06-19 11:05:56 on dist-test-slave-8vcz"
server_key: "59493604a8b65a61da287b21cc7d2da6"
server_key_iv: "75afb687929600f1f2b5b9bb5d121ac4"
server_key_version: "encryptionkey@0"
I20260619 11:05:56.127555 27086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260619 11:05:56.130791 27101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:56.132097 27086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:05:56.132229 27086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "8fa2cd2670944b70a3569598c5031a5e"
format_stamp: "Formatted at 2026-06-19 11:05:56 on dist-test-slave-8vcz"
server_key: "59493604a8b65a61da287b21cc7d2da6"
server_key_iv: "75afb687929600f1f2b5b9bb5d121ac4"
server_key_version: "encryptionkey@0"
I20260619 11:05:56.132347 27086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:56.147588 27086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:56.150286 27086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:56.150486 27086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:56.151036 27086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:56.151889 27086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:56.151958 27086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:56.152030 27086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:56.152082 27086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:56.160511 27086 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:40753
I20260619 11:05:56.160529 27214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:40753 every 8 connection(s)
I20260619 11:05:56.161331 27086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:05:56.162035 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 27086
I20260619 11:05:56.162156 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:05:56.162431 16628 external_mini_cluster.cc:1428] Setting key 73631c2e829c704bf002510be657078c
Jun 19 11:05:56 dist-test-slave-8vcz krb5kdc[26845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867156, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:05:56.175889 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.163032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:41911 (local address 127.16.61.62:46877)
0619 11:05:56.163166 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.163170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.164117 (+ 947us) server_negotiation.cc:407] Connection header received
0619 11:05:56.166116 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.166120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.166170 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.166248 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:56.167327 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.168214 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.169396 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.169825 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.169966 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.172426 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:56.172429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:56.172431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:56.172453 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:56.173921 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:56.174372 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:56.174377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:56.174379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:56.174463 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:56.174790 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:56.174794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:56.174795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:56.175017 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:56.175121 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.175581 (+ 460us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.175721 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:05:56.176606 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.163478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46877 (local address 127.16.61.2:41911)
0619 11:05:56.163976 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:56.164031 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:56.165926 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:56.166406 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:56.166413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:56.166746 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:56.167179 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.167192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.168334 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.168337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.169275 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.169284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.169965 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.169968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.170110 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.170671 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:56.170686 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:56.172265 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:56.174019 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:56.174023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:56.174033 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:56.174261 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:56.174554 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:56.174559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:56.174561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:56.174689 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:56.175117 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:56.175123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:56.175348 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 11:05:56.175526 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 11:05:56.175838 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":141,"threads_started":1}
I20260619 11:05:56.177790 27215 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46877
I20260619 11:05:56.178025 27215 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:56.178479 27215 heartbeater.cc:511] Master 127.16.61.62:46877 requested a full tablet report, sending...
I20260619 11:05:56.179534 26890 ts_manager.cc:194] Registered new tserver with Master: 8fa2cd2670944b70a3569598c5031a5e (127.16.61.2:40753)
I20260619 11:05:56.180096 26890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:41911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:05:56.190374 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46877
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:05:56.296077 27220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:05:56.296329 27220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:05:56.296398 27220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:05:56.300256 27220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:05:56.300339 27220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:05:56.300426 27220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:05:56.305361 27220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:05:56.306298 27220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:05:56.307224 27220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:05:56.316155 27220 server_base.cc:1034] running on GCE node
W20260619 11:05:56.316097 27225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:56.316128 27226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:05:56.316097 27228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:05:56.316808 27220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:05:56.317373 27220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:05:56.318521 27220 hybrid_clock.cc:648] HybridClock initialized: now 1781867156318492 us; error 43 us; skew 500 ppm
Jun 19 11:05:56 dist-test-slave-8vcz krb5kdc[26845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867156, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:05:56.321110 27220 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:05:56.321949 27220 webserver.cc:460] Webserver started at http://127.16.61.3:42229/ using document root <none> and password file <none>
I20260619 11:05:56.322487 27220 fs_manager.cc:359] Metadata directory not provided
I20260619 11:05:56.322575 27220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:05:56.322794 27220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:05:56.324744 27220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "75a4f629c65c46fdaf5c1132ad22eb0e"
format_stamp: "Formatted at 2026-06-19 11:05:56 on dist-test-slave-8vcz"
server_key: "8427c509b3ab72eeb44bb93e2de7aa24"
server_key_iv: "4064282874730bc35a055e5afe32ec21"
server_key_version: "encryptionkey@0"
I20260619 11:05:56.325270 27220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "75a4f629c65c46fdaf5c1132ad22eb0e"
format_stamp: "Formatted at 2026-06-19 11:05:56 on dist-test-slave-8vcz"
server_key: "8427c509b3ab72eeb44bb93e2de7aa24"
server_key_iv: "4064282874730bc35a055e5afe32ec21"
server_key_version: "encryptionkey@0"
I20260619 11:05:56.328852 27220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:05:56.331089 27235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:56.332276 27220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:05:56.332403 27220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "75a4f629c65c46fdaf5c1132ad22eb0e"
format_stamp: "Formatted at 2026-06-19 11:05:56 on dist-test-slave-8vcz"
server_key: "8427c509b3ab72eeb44bb93e2de7aa24"
server_key_iv: "4064282874730bc35a055e5afe32ec21"
server_key_version: "encryptionkey@0"
I20260619 11:05:56.332522 27220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:05:56.353278 27220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:05:56.355979 27220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:05:56.356170 27220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:05:56.356721 27220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:05:56.357507 27220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:05:56.357575 27220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:56.357642 27220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:05:56.357693 27220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:05:56.366191 27220 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:36333
I20260619 11:05:56.366218 27348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:36333 every 8 connection(s)
I20260619 11:05:56.367002 27220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:05:56.376240 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 27220
I20260619 11:05:56.376441 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:05:56.376741 16628 external_mini_cluster.cc:1428] Setting key ae0def23998158c49e61931407cd800e
Jun 19 11:05:56 dist-test-slave-8vcz krb5kdc[26845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867156, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:05:56.382000 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.368749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:58075 (local address 127.16.61.62:46877)
0619 11:05:56.368876 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.368880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.369693 (+ 813us) server_negotiation.cc:407] Connection header received
0619 11:05:56.370973 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.370975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.371020 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.371092 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:56.372312 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.373234 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.375206 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.375516 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.375618 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.378709 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:56.378713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:56.378715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:56.378744 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:56.380283 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:56.380723 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:56.380725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:56.380727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:56.380796 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:56.381089 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:56.381091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:56.381092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:56.381242 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:56.381315 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.381777 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.381873 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:05:56.382728 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.369079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46877 (local address 127.16.61.3:58075)
0619 11:05:56.369503 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:56.369543 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:56.370778 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:56.371245 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:56.371251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:56.371662 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:05:56.372165 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.372178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.373380 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.373383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.375041 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.375052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.375700 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.375706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.375895 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.376828 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:05:56.376848 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:05:56.378552 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:05:56.380380 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:56.380385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:56.380396 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:56.380638 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:56.380881 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:05:56.380883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:05:56.380884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:05:56.381013 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:05:56.381323 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:05:56.381328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:05:56.381536 (+ 208us) client_negotiation.cc:769] Sending connection context
0619 11:05:56.381688 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 11:05:56.381915 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260619 11:05:56.383728 27349 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46877
I20260619 11:05:56.383920 27349 heartbeater.cc:464] Registering TS with master...
I20260619 11:05:56.384316 27349 heartbeater.cc:511] Master 127.16.61.62:46877 requested a full tablet report, sending...
I20260619 11:05:56.385185 26890 ts_manager.cc:194] Registered new tserver with Master: 75a4f629c65c46fdaf5c1132ad22eb0e (127.16.61.3:36333)
I20260619 11:05:56.385716 26890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:58075
I20260619 11:05:56.391273 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:05:56.399950 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.393327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.16.61.62:46877)
0619 11:05:56.393432 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.393435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.393647 (+ 212us) server_negotiation.cc:407] Connection header received
0619 11:05:56.393785 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.393787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.393839 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.393914 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:05:56.394543 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.395063 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.395940 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.396134 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.396208 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.397465 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:05:56.397467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:05:56.397469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:05:56.397484 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:05:56.398609 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:56.398958 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:56.398961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:56.398962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:56.399028 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:05:56.399242 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:05:56.399244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:05:56.399245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:05:56.399420 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:05:56.399528 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.399769 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.399848 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:05:56.402539 26890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:05:56.405494 26890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:05:56.419260 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.413242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:41433 (local address 127.0.0.1:51014)
0619 11:05:56.413492 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:56.413527 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:56.413636 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:56.415242 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:56.415245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:56.415262 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:56.415349 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.415354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.416782 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.416785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.417935 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.417942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.418717 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.418720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.418825 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.418864 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 11:05:56.418930 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 11:05:56.419092 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":28,"spinlock_wait_cycles":2560,"thread_start_us":95,"threads_started":1}
I20260619 11:05:56.419261 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.413157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:36333 (local address 127.0.0.1:50976)
0619 11:05:56.413469 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:56.413511 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:56.413636 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:56.414722 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:56.414728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:56.414753 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:56.414870 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.414876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.416270 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.416273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.417670 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.417679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.418773 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.418777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.418855 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.418877 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:05:56.418942 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:05:56.419087 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":110,"spinlock_wait_cycles":7936,"thread_start_us":96,"threads_started":1}
I20260619 11:05:56.419484 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.413834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50976 (local address 127.16.61.3:36333)
0619 11:05:56.414238 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.414250 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.414272 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:05:56.414342 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.414347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.414562 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.414719 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:56.414986 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.416138 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.417787 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.418617 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.419043 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.419181 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.419229 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.419317 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":100,"threads_started":1}
I20260619 11:05:56.419541 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.414403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.16.61.2:40753)
0619 11:05:56.414712 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.414724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.414744 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:05:56.414810 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.414814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.415049 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.415186 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:56.415523 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.416364 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.417940 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.418679 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.418772 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.418925 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.419323 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.419404 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
I20260619 11:05:56.419741 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.413901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:40753 (local address 127.0.0.1:50460)
0619 11:05:56.414151 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:56.414162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:56.414238 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:56.415331 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:56.415333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:56.415347 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:56.415418 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.415422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.416507 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.416510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.417822 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.417832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.418982 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.418987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.419091 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.419110 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 11:05:56.419173 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 11:05:56.419240 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":23,"thread_start_us":86,"threads_started":1}
I20260619 11:05:56.420339 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.414102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.16.61.1:41433)
0619 11:05:56.414779 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.414790 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.414810 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:05:56.414880 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.414884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.415094 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.415484 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:56.415617 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.416640 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.418063 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.418563 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.420039 (+ 1476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.420134 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.420170 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.420231 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":95,"threads_started":1}
I20260619 11:05:56.422564 27015 tablet_service.cc:1467] Processing CreateTablet for tablet eb688a022cd14e6e966567e83736da81 (DEFAULT_TABLE table=test-table [id=3ef41edb3738405a92e648ffaf3bd1d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:56.422629 27283 tablet_service.cc:1467] Processing CreateTablet for tablet eb688a022cd14e6e966567e83736da81 (DEFAULT_TABLE table=test-table [id=3ef41edb3738405a92e648ffaf3bd1d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:56.422631 27149 tablet_service.cc:1467] Processing CreateTablet for tablet eb688a022cd14e6e966567e83736da81 (DEFAULT_TABLE table=test-table [id=3ef41edb3738405a92e648ffaf3bd1d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:05:56.423520 27015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb688a022cd14e6e966567e83736da81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:56.423581 27283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb688a022cd14e6e966567e83736da81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:56.423576 27149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb688a022cd14e6e966567e83736da81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:05:56.429637 27366 tablet_bootstrap.cc:492] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043: Bootstrap starting.
I20260619 11:05:56.430680 27368 tablet_bootstrap.cc:492] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Bootstrap starting.
I20260619 11:05:56.431224 27367 tablet_bootstrap.cc:492] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Bootstrap starting.
I20260619 11:05:56.432260 27366 tablet_bootstrap.cc:654] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:56.432693 27368 tablet_bootstrap.cc:654] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:56.433219 27367 tablet_bootstrap.cc:654] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Neither blocks nor log segments found. Creating new log.
I20260619 11:05:56.433609 27366 log.cc:824] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:56.433842 27368 log.cc:824] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:56.433893 27367 log.cc:824] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:05:56.435609 27366 tablet_bootstrap.cc:492] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043: No bootstrap required, opened a new log
I20260619 11:05:56.435748 27368 tablet_bootstrap.cc:492] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: No bootstrap required, opened a new log
I20260619 11:05:56.435865 27366 ts_tablet_manager.cc:1397] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:05:56.435917 27368 ts_tablet_manager.cc:1397] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:05:56.438728 27368 raft_consensus.cc:348] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.438730 27366 raft_consensus.cc:348] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.438922 27368 raft_consensus.cc:374] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:56.438962 27366 raft_consensus.cc:374] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:56.439008 27368 raft_consensus.cc:729] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a4f629c65c46fdaf5c1132ad22eb0e, State: Initialized, Role: FOLLOWER
I20260619 11:05:56.439011 27366 raft_consensus.cc:729] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2dec33e86c4711ab48d1c229616043, State: Initialized, Role: FOLLOWER
I20260619 11:05:56.439404 27366 consensus_queue.cc:260] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.439399 27368 consensus_queue.cc:260] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.440215 27349 heartbeater.cc:503] Master 127.16.61.62:46877 was elected leader, sending a full tablet report...
I20260619 11:05:56.440215 27081 heartbeater.cc:503] Master 127.16.61.62:46877 was elected leader, sending a full tablet report...
I20260619 11:05:56.440405 27366 ts_tablet_manager.cc:1428] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260619 11:05:56.440943 27367 tablet_bootstrap.cc:492] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: No bootstrap required, opened a new log
I20260619 11:05:56.441129 27367 ts_tablet_manager.cc:1397] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260619 11:05:56.441439 27368 ts_tablet_manager.cc:1428] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260619 11:05:56.445158 27367 raft_consensus.cc:348] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.445340 27367 raft_consensus.cc:374] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:05:56.445434 27367 raft_consensus.cc:729] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa2cd2670944b70a3569598c5031a5e, State: Initialized, Role: FOLLOWER
I20260619 11:05:56.445806 27367 consensus_queue.cc:260] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.446358 27215 heartbeater.cc:503] Master 127.16.61.62:46877 was elected leader, sending a full tablet report...
I20260619 11:05:56.446616 27367 ts_tablet_manager.cc:1428] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 11:05:56.451794 27082 tablet.cc:2369] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:56.618006 27350 tablet.cc:2369] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:05:56.662427 27216 tablet.cc:2369] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:05:56.679000 27372 raft_consensus.cc:482] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:05:56.679162 27372 raft_consensus.cc:504] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.680332 27372 leader_election.cc:288] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a2dec33e86c4711ab48d1c229616043 (127.16.61.1:41433), 8fa2cd2670944b70a3569598c5031a5e (127.16.61.2:40753)
I20260619 11:05:56.684863 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.680587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:40753 (local address 127.16.61.3:46405)
0619 11:05:56.680703 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:56.680719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:56.680825 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:56.681151 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:56.681154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:56.681176 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:56.681303 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.681314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.682358 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.682360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.683558 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.683567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.684414 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.684416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.684546 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.684571 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:05:56.684648 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 11:05:56.684727 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 11:05:56.684875 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.680590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:41433 (local address 127.16.61.3:47217)
0619 11:05:56.681260 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:56.681274 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:05:56.681379 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:05:56.681634 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:05:56.681636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:05:56.681649 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:05:56.681735 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.681739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.682588 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.682591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.683726 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:05:56.683733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.684516 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.684519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:05:56.684613 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.684642 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 11:05:56.684705 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 11:05:56.684751 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":339,"thread_start_us":118,"threads_started":1}
I20260619 11:05:56.684895 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.680683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:47217 (local address 127.16.61.1:41433)
0619 11:05:56.680796 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.680799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.681427 (+ 628us) server_negotiation.cc:407] Connection header received
0619 11:05:56.681484 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.681487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.681534 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.681620 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:56.681838 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.682490 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.683833 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.684404 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.684500 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.684555 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.684734 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.684794 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 11:05:56.684895 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.680651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:46405 (local address 127.16.61.2:40753)
0619 11:05:56.680774 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.680778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.680793 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:05:56.680956 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.680959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.681015 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.681098 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:05:56.681447 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.682256 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.683700 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.684297 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.684400 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.684481 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.684696 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.684769 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:05:56.685465 27035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb688a022cd14e6e966567e83736da81" candidate_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a2dec33e86c4711ab48d1c229616043" is_pre_election: true
I20260619 11:05:56.685465 27169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb688a022cd14e6e966567e83736da81" candidate_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa2cd2670944b70a3569598c5031a5e" is_pre_election: true
I20260619 11:05:56.685745 27035 raft_consensus.cc:2454] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a4f629c65c46fdaf5c1132ad22eb0e in term 0.
I20260619 11:05:56.685745 27169 raft_consensus.cc:2454] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a4f629c65c46fdaf5c1132ad22eb0e in term 0.
I20260619 11:05:56.686211 27237 leader_election.cc:302] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75a4f629c65c46fdaf5c1132ad22eb0e, 8fa2cd2670944b70a3569598c5031a5e; no voters:
I20260619 11:05:56.686481 27372 raft_consensus.cc:2781] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:05:56.686574 27372 raft_consensus.cc:482] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:05:56.686631 27372 raft_consensus.cc:3037] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:56.687611 27372 raft_consensus.cc:504] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.688009 27372 leader_election.cc:288] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [CANDIDATE]: Term 1 election: Requested vote from peers 1a2dec33e86c4711ab48d1c229616043 (127.16.61.1:41433), 8fa2cd2670944b70a3569598c5031a5e (127.16.61.2:40753)
I20260619 11:05:56.688339 27035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb688a022cd14e6e966567e83736da81" candidate_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a2dec33e86c4711ab48d1c229616043"
I20260619 11:05:56.688444 27169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb688a022cd14e6e966567e83736da81" candidate_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fa2cd2670944b70a3569598c5031a5e"
I20260619 11:05:56.688484 27035 raft_consensus.cc:3037] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:56.688584 27169 raft_consensus.cc:3037] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:05:56.689476 27035 raft_consensus.cc:2454] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a4f629c65c46fdaf5c1132ad22eb0e in term 1.
I20260619 11:05:56.689625 27169 raft_consensus.cc:2454] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a4f629c65c46fdaf5c1132ad22eb0e in term 1.
I20260619 11:05:56.689740 27238 leader_election.cc:302] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a2dec33e86c4711ab48d1c229616043, 75a4f629c65c46fdaf5c1132ad22eb0e; no voters:
I20260619 11:05:56.689898 27372 raft_consensus.cc:2781] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:05:56.690115 27372 raft_consensus.cc:686] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 1 LEADER]: Becoming Leader. State: Replica: 75a4f629c65c46fdaf5c1132ad22eb0e, State: Running, Role: LEADER
I20260619 11:05:56.690403 27372 consensus_queue.cc:237] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } }
I20260619 11:05:56.693624 26890 catalog_manager.cc:5495] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 75a4f629c65c46fdaf5c1132ad22eb0e (127.16.61.3). New cstate: current_term: 1 leader_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75a4f629c65c46fdaf5c1132ad22eb0e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 } health_report { overall_health: UNKNOWN } } }
I20260619 11:05:56.706784 27362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.703624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50984 (local address 127.16.61.3:36333)
0619 11:05:56.703736 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:05:56.703742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:05:56.703790 (+ 48us) server_negotiation.cc:407] Connection header received
0619 11:05:56.703940 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:05:56.703942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:05:56.703996 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:05:56.704073 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:05:56.704279 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.704734 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.705533 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:05:56.705721 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:05:56.705790 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:05:56.706071 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:05:56.706417 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:05:56.706509 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:05:56.706598 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 11:05:56.706638 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 11:05:56.711978 27169 raft_consensus.cc:1261] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 1 FOLLOWER]: Refusing update from remote peer 75a4f629c65c46fdaf5c1132ad22eb0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:56.712046 27035 raft_consensus.cc:1261] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 1 FOLLOWER]: Refusing update from remote peer 75a4f629c65c46fdaf5c1132ad22eb0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:05:56.712678 27372 consensus_queue.cc:1035] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a2dec33e86c4711ab48d1c229616043" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:56.712853 27376 consensus_queue.cc:1035] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fa2cd2670944b70a3569598c5031a5e" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:05:56.719043 27379 mvcc.cc:205] Tried to move back new op lower bound from 7298527873885880320 to 7298527873806245888. Current Snapshot: MvccSnapshot[applied={T|T < 7298527873885880320}]
I20260619 11:05:56.719220 27381 mvcc.cc:205] Tried to move back new op lower bound from 7298527873885880320 to 7298527873806245888. Current Snapshot: MvccSnapshot[applied={T|T < 7298527873885880320}]
I20260619 11:05:56.721261 27380 mvcc.cc:205] Tried to move back new op lower bound from 7298527873885880320 to 7298527873806245888. Current Snapshot: MvccSnapshot[applied={T|T < 7298527873885880320}]
I20260619 11:05:56.728483 26890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:56.728607 26890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:05:56.730782 26890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c638916fc54a4cd3a674ce83edbb0442: Sending DeleteTablet for 3 replicas of tablet eb688a022cd14e6e966567e83736da81
I20260619 11:05:56.731492 27015 tablet_service.cc:1514] Processing DeleteTablet for tablet eb688a022cd14e6e966567e83736da81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:56 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:51014
I20260619 11:05:56.731529 27149 tablet_service.cc:1514] Processing DeleteTablet for tablet eb688a022cd14e6e966567e83736da81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:56 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:50460
I20260619 11:05:56.731962 27388 tablet_replica.cc:330] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043: stopping tablet replica
I20260619 11:05:56.732156 27283 tablet_service.cc:1514] Processing DeleteTablet for tablet eb688a022cd14e6e966567e83736da81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:05:56 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:50976
I20260619 11:05:56.732290 27389 tablet_replica.cc:330] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: stopping tablet replica
I20260619 11:05:56.732388 27388 raft_consensus.cc:2229] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:56.732549 27389 raft_consensus.cc:2229] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:05:56.732678 27388 raft_consensus.cc:2258] T eb688a022cd14e6e966567e83736da81 P 1a2dec33e86c4711ab48d1c229616043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:56.732698 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 26952
I20260619 11:05:56.732785 27389 raft_consensus.cc:2258] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:56.734184 27389 ts_tablet_manager.cc:1905] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:56.737159 27389 ts_tablet_manager.cc:1918] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:56.737303 27389 log.cc:1196] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/eb688a022cd14e6e966567e83736da81
I20260619 11:05:56.737640 27389 ts_tablet_manager.cc:1939] T eb688a022cd14e6e966567e83736da81 P 8fa2cd2670944b70a3569598c5031a5e: Deleting consensus metadata
I20260619 11:05:56.738649 26875 catalog_manager.cc:4841] TS 8fa2cd2670944b70a3569598c5031a5e (127.16.61.2:40753): tablet eb688a022cd14e6e966567e83736da81 (table test-table [id=3ef41edb3738405a92e648ffaf3bd1d0]) successfully deleted
W20260619 11:05:56.740722 26876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:41433 (error 108)
I20260619 11:05:56.740854 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 27086
I20260619 11:05:56.741325 27390 tablet_replica.cc:330] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: stopping tablet replica
I20260619 11:05:56.741608 27390 raft_consensus.cc:2229] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:05:56.742251 27363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:05:56.741933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:41433 (local address 127.0.0.1:51016)
0619 11:05:56.742105 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 11:05:56.742174 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:41433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 11:05:56.742424 26876 catalog_manager.cc:4584] TS 1a2dec33e86c4711ab48d1c229616043 (127.16.61.1:41433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb688a022cd14e6e966567e83736da81: Network error: Client connection negotiation failed: client connection to 127.16.61.1:41433: connect: Connection refused (error 111)
I20260619 11:05:56.743000 27390 raft_consensus.cc:2258] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:05:56.744432 27390 ts_tablet_manager.cc:1905] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:05:56.746939 27390 ts_tablet_manager.cc:1918] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:05:56.746984 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 27220
I20260619 11:05:56.747040 27390 log.cc:1196] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/eb688a022cd14e6e966567e83736da81
I20260619 11:05:56.747341 27390 ts_tablet_manager.cc:1939] T eb688a022cd14e6e966567e83736da81 P 75a4f629c65c46fdaf5c1132ad22eb0e: Deleting consensus metadata
I20260619 11:05:56.751745 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 26858
2026-06-19T11:05:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2403 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:05:56 dist-test-slave-8vcz krb5kdc[27394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:05:56 dist-test-slave-8vcz krb5kdc[27394](info): set up 2 sockets
Jun 19 11:05:56 dist-test-slave-8vcz krb5kdc[27394](info): commencing operation
krb5kdc: starting...
W20260619 11:05:57.788602 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:05:57 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:05:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/rangeradmin_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/rangeradmin_127.16.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/rangerlookup_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/rangerlookup_127.16.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/HTTP_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/HTTP_127.16.61.20@KRBTEST.COM.keytab.
I20260619 11:05:57.943868 16628 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIllhcy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/postgres -l logfile start
2026-06-19 11:06:00.281 UTC [27424] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 11:06:00.281 UTC [27424] LOG: listening on IPv4 address "127.16.61.20", port 36813
2026-06-19 11:06:00.285 UTC [27424] LOG: listening on Unix socket "/tmp/.s.PGSQL.36813"
2026-06-19 11:06:00.304 UTC [27427] LOG: database system was shut down at 2026-06-19 11:05:58 UTC
2026-06-19 11:06:00.308 UTC [27424] LOG: database system is ready to accept connections
I20260619 11:06:01.256405 16628 mini_postgres.cc:96] Postgres bound to 36813
2026-06-19 11:06:01.262 UTC [27435] FATAL: database "slave" does not exist
127.16.61.20:36813 - accepting connections
I20260619 11:06:01.263307 16628 mini_ranger.cc:161] Starting Ranger...
I20260619 11:06:01.277439 16628 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 11:06:01.712798 16628 mini_ranger.cc:87] Created ranger Postgres database
I20260619 11:06:01.712893 16628 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-admin
2026-06-19 11:06:01,987 [I] DB FLAVOR :POSTGRES
2026-06-19 11:06:01,987 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:06:01,987 [I] Checking connection..
2026-06-19 11:06:01,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 11:06:02,229 [I] Checking connection passed.
2026-06-19 11:06:02,229 [I] --------- Verifying version history table ---------
2026-06-19 11:06:02,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:06:02,489 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:06:02,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:06:02,740 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:06:02,740 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 11:06:02,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 11:06:02.955 UTC [27499] WARNING: there is no transaction in progress
2026-06-19 11:06:02,961 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 11:06:02,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:06:03,201 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 11:06:03,201 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:06:03,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 11:06:03,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 11:06:03,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8vcz.c.gcp-upstream.internal','N') ;"
2026-06-19 11:06:03,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:06:04,108 [I] Table x_portal_user does not exist in database ranger
2026-06-19 11:06:04,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:06:04,333 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 11:06:04,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:06:04,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:06:04,813 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 11:06:04,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 11:06:05.718 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.733 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.748 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.762 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.779 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.794 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.843 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.849 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.862 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.871 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.881 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.894 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.903 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.909 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.915 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:05.922 UTC [27607] WARNING: there is no transaction in progress
2026-06-19 11:06:06,500 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 11:06:06,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 11:06:06,731 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 11:06:06,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:06:06,965 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 11:06:06,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:06:07,185 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 11:06:07,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:06:07,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:06:07,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8vcz.c.gcp-upstream.internal';"
2026-06-19 11:06:07,834 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 11:06:07,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 11:06:08,055 [I] DB_PATCHES have already been applied
I20260619 11:06:08.061393 16628 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIllhcy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 11:06:08.061448 16628 mini_ranger.cc:193] Using host: 127.16.61.20
I20260619 11:06:08.064857 16628 mini_ranger.cc:227] Ranger admin URL: http://127.16.61.20:43185
Jun 19, 2026 11:06:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:06:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:06:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 11:06:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 11:06:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 11:06:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 11:06:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 11:06:09.177528 16628 mini_ranger.cc:160] Time spent starting Ranger: real 7.914s user 0.007s sys 0.002s
Jun 19, 2026 11:06:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 11:06:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 11:06:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 11:06:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 11:06:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 11:06:30.708868 16628 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/rangerkms_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/rangerkms_127.16.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.61.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.61.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.61.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/HTTP_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.61.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/HTTP_127.16.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 11:06:30 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867190, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 11:06:30.788226 16628 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 11:06:30.797072 16628 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 11:06:31.594565 16628 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 11:06:31.594655 16628 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms
I20260619 11:06:31.594745 16628 mini_ranger_kms.cc:227] Using postgres at 127.16.61.20:36813
2026-06-19 11:06:31,808 [I] DB FLAVOR :POSTGRES
2026-06-19 11:06:31,808 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:06:31,808 [I] Checking connection
2026-06-19 11:06:31,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 11:06:32,067 [I] connection success
2026-06-19 11:06:32,068 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 11:06:32,068 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 11:06:32,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 11:06:32,297 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 11:06:32,297 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:06:32,298 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 11:06:32,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:36813/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 11:06:32,556 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 11:06:32.970827 16628 mini_ranger_kms.cc:310] Created kms service
I20260619 11:06:33.156703 16628 mini_ranger_kms.cc:326] Created kudu user
I20260619 11:06:33.196534 16628 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 11:06:33 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867190, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:06:33.564853 16628 mini_ranger_kms.cc:384] Added ranger policy
I20260619 11:06:33.565011 16628 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 11:06:33.565037 16628 mini_ranger_kms.cc:240] Using host: 127.16.61.20
I20260619 11:06:33.569221 16628 mini_ranger_kms.cc:276] Ranger KMS PID: 27787
I20260619 11:06:33.569343 16628 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.61.20:53789
Jun 19, 2026 11:06:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:06:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:06:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 11:06:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 11:06:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 11:06:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 11:06:35.831774 16628 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.044s user 0.022s sys 0.174s
I20260619 11:06:35.831909 16628 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 11:06:35.832011 16628 mini_ranger_kms.cc:401] 127.16.61.20:53789/kms/v1/keys
Jun 19 11:06:35 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867195, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.61.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 11:06:36 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867195, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.61.20@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
Jun 19 11:06:36 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867195, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.61.20@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
Jun 19 11:06:38 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:06:38.816421 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:43325
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:43325
--ranger_config_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:53789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:06:38.924396 27838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:06:38.924638 27838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:06:38.924678 27838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:06:38.928642 27838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:06:38.928697 27838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:06:38.928716 27838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:06:38.928732 27838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:06:38.928747 27838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:06:38.934170 27838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.61.20:53789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:43325
--ranger_config_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:06:38.935163 27838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:06:38.936177 27838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:06:38.945587 27838 server_base.cc:1034] running on GCE node
W20260619 11:06:38.945611 27843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:06:38.945817 27846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:06:38.945930 27844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:06:38.946516 27838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:06:38.947572 27838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:06:38.948746 27838 hybrid_clock.cc:648] HybridClock initialized: now 1781867198948727 us; error 33 us; skew 500 ppm
Jun 19 11:06:38 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:06:38.951586 27838 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:06:38.952591 27838 webserver.cc:460] Webserver started at http://127.16.61.62:46679/ using document root <none> and password file <none>
I20260619 11:06:38.953212 27838 fs_manager.cc:359] Metadata directory not provided
I20260619 11:06:38.953284 27838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:06:38.953485 27838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:06:38 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867198, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:06:39.004712 27838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "788d3445b4e3446ead9e15aac2efb02a"
format_stamp: "Formatted at 2026-06-19 11:06:38 on dist-test-slave-8vcz"
server_key: "936843f089573e540de7fd023e6d7a85"
server_key_iv: "f203461d2a2ac758c382be30bc88ed89"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:39.005334 27838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "788d3445b4e3446ead9e15aac2efb02a"
format_stamp: "Formatted at 2026-06-19 11:06:38 on dist-test-slave-8vcz"
server_key: "936843f089573e540de7fd023e6d7a85"
server_key_iv: "f203461d2a2ac758c382be30bc88ed89"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:39.009230 27838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 11:06:39.025337 27853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:39.026679 27838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 11:06:39.026782 27838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "788d3445b4e3446ead9e15aac2efb02a"
format_stamp: "Formatted at 2026-06-19 11:06:38 on dist-test-slave-8vcz"
server_key: "936843f089573e540de7fd023e6d7a85"
server_key_iv: "f203461d2a2ac758c382be30bc88ed89"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:39.026925 27838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:06:39.038297 27838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:06:39.041159 27838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:06:39.041348 27838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:06:39.049327 27838 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:43325
I20260619 11:06:39.049379 27905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:43325 every 8 connection(s)
I20260619 11:06:39.050256 27838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:06:39.052836 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 27838
I20260619 11:06:39.052991 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:06:39.053298 16628 external_mini_cluster.cc:1428] Setting key b94269daa37d147e27cdd728144750af
I20260619 11:06:39.053630 27906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:06:39.060901 27906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a: Bootstrap starting.
Jun 19 11:06:39 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:06:39.063876 27906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a: Neither blocks nor log segments found. Creating new log.
I20260619 11:06:39.065129 27906 log.cc:824] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a: Log is configured to *not* fsync() on all Append() calls
I20260619 11:06:39.066946 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:39.054705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.16.61.62:43325)
0619 11:06:39.055090 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:39.055107 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:39.055152 (+ 45us) server_negotiation.cc:407] Connection header received
0619 11:06:39.055871 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:39.055893 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:39.056193 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:39.056508 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:06:39.057258 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:39.058336 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:39.059161 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:39.059627 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:39.059877 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:39.062398 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:06:39.062403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:06:39.062416 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:06:39.062441 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:06:39.064151 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:39.064585 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:39.064596 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:39.064600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:39.064706 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:39.064995 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:39.065000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:39.065002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:39.065434 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:06:39.065676 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:39.065999 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 11:06:39.066286 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":135,"threads_started":1}
I20260619 11:06:39.067538 27906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a: No bootstrap required, opened a new log
I20260619 11:06:39.071281 27906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d3445b4e3446ead9e15aac2efb02a" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43325 } }
I20260619 11:06:39.071530 27906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:06:39.071576 27906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788d3445b4e3446ead9e15aac2efb02a, State: Initialized, Role: FOLLOWER
I20260619 11:06:39.072170 27906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d3445b4e3446ead9e15aac2efb02a" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43325 } }
I20260619 11:06:39.072343 27906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:06:39.072412 27906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:06:39.072508 27906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:06:39.073524 27906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d3445b4e3446ead9e15aac2efb02a" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43325 } }
I20260619 11:06:39.073886 27906 leader_election.cc:302] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788d3445b4e3446ead9e15aac2efb02a; no voters:
I20260619 11:06:39.074235 27906 leader_election.cc:288] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:06:39.074335 27911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:06:39.074590 27911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [term 1 LEADER]: Becoming Leader. State: Replica: 788d3445b4e3446ead9e15aac2efb02a, State: Running, Role: LEADER
I20260619 11:06:39.074918 27911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d3445b4e3446ead9e15aac2efb02a" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43325 } }
I20260619 11:06:39.075443 27906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:06:39.076653 27912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788d3445b4e3446ead9e15aac2efb02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d3445b4e3446ead9e15aac2efb02a" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43325 } } }
I20260619 11:06:39.076786 27912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [sys.catalog]: This master's current role is: LEADER
I20260619 11:06:39.077087 27913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788d3445b4e3446ead9e15aac2efb02a. Latest consensus state: current_term: 1 leader_uuid: "788d3445b4e3446ead9e15aac2efb02a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d3445b4e3446ead9e15aac2efb02a" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43325 } } }
I20260619 11:06:39.077159 27913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a [sys.catalog]: This master's current role is: LEADER
I20260619 11:06:39.077613 27906 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:06:39.077962 27920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:06:39.085947 27920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:06:39.093715 27920 catalog_manager.cc:1347] Generated new cluster ID: fc64d41559084fee8f7ffe2138a6b6f0
I20260619 11:06:39.093779 27920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:06:39.110577 27920 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:06:39.112080 27920 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:06:39.123338 27920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a: Generated new TSK 0
I20260619 11:06:39.124284 27920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:06:39.812836 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:53789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43325
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:06:39.927062 27951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:06:39.927340 27951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:06:39.927448 27951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:06:39.931721 27951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:06:39.931789 27951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:06:39.931877 27951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:06:39.937412 27951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.61.20:53789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:06:39.938455 27951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:06:39.939530 27951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:06:39.950199 27956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:06:39.950157 27957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:06:39.950373 27951 server_base.cc:1034] running on GCE node
W20260619 11:06:39.950157 27959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:06:39.950965 27951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:06:39.951596 27951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:06:39.952764 27951 hybrid_clock.cc:648] HybridClock initialized: now 1781867199952732 us; error 51 us; skew 500 ppm
Jun 19 11:06:39 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867199, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:06:39.955453 27951 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:06:39.956301 27951 webserver.cc:460] Webserver started at http://127.16.61.1:43221/ using document root <none> and password file <none>
I20260619 11:06:39.956843 27951 fs_manager.cc:359] Metadata directory not provided
I20260619 11:06:39.956933 27951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:06:39.957176 27951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:06:39 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867199, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:06:39.975221 27951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "67926f00bb464bb596118e62f02ddddc"
format_stamp: "Formatted at 2026-06-19 11:06:39 on dist-test-slave-8vcz"
server_key: "b6cf8846682b296a3c279c9bac3ab644"
server_key_iv: "597ec8da4534263110dc117a006ccae9"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:39.975873 27951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "67926f00bb464bb596118e62f02ddddc"
format_stamp: "Formatted at 2026-06-19 11:06:39 on dist-test-slave-8vcz"
server_key: "b6cf8846682b296a3c279c9bac3ab644"
server_key_iv: "597ec8da4534263110dc117a006ccae9"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:39.981887 27951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260619 11:06:39.996865 27968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:39.998440 27951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 11:06:39.998570 27951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "67926f00bb464bb596118e62f02ddddc"
format_stamp: "Formatted at 2026-06-19 11:06:39 on dist-test-slave-8vcz"
server_key: "b6cf8846682b296a3c279c9bac3ab644"
server_key_iv: "597ec8da4534263110dc117a006ccae9"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:39.998692 27951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:06:40.029884 27951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:06:40.033052 27951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:06:40.033262 27951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:06:40.033906 27951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:06:40.034749 27951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:06:40.034819 27951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:40.034917 27951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:06:40.034953 27951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:40.045900 27951 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:42995
I20260619 11:06:40.045929 28081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:42995 every 8 connection(s)
I20260619 11:06:40.046849 27951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:06:40.051617 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 27951
I20260619 11:06:40.051759 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:06:40.052064 16628 external_mini_cluster.cc:1428] Setting key 9ce5a26c42010340160db6b186109c6e
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867199, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:06:40.063612 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.048752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:48139 (local address 127.16.61.62:43325)
0619 11:06:40.049031 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:40.049040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:40.049858 (+ 818us) server_negotiation.cc:407] Connection header received
0619 11:06:40.050813 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:40.050816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:40.050885 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:40.050973 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:06:40.052403 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.053172 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.054871 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.055286 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.055372 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.058476 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:06:40.058481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:06:40.058486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:06:40.058516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:06:40.061027 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:40.061701 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:40.061706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:40.061708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:40.061796 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:40.062213 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:40.062218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:40.062223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:40.062469 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:06:40.062548 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:40.063275 (+ 727us) server_negotiation.cc:299] Negotiation successful
0619 11:06:40.063427 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:06:40.065462 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.049261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43325 (local address 127.16.61.1:48139)
0619 11:06:40.049709 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:40.049761 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:06:40.050611 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:06:40.051159 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:06:40.051165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:06:40.051515 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:06:40.052217 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.052237 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.053321 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.053324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.054728 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.054743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.055528 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.055530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.055725 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.056458 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:06:40.056483 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:06:40.058264 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:06:40.061187 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:06:40.061196 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:06:40.061214 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:06:40.061578 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:06:40.061914 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:06:40.061918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:06:40.061920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:06:40.062094 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:06:40.062633 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:06:40.062640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:06:40.062967 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 11:06:40.064127 (+ 1160us) client_negotiation.cc:240] Negotiation successful
0619 11:06:40.064451 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
I20260619 11:06:40.067094 28082 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43325
I20260619 11:06:40.067322 28082 heartbeater.cc:464] Registering TS with master...
I20260619 11:06:40.067916 28082 heartbeater.cc:511] Master 127.16.61.62:43325 requested a full tablet report, sending...
I20260619 11:06:40.069545 27870 ts_manager.cc:194] Registered new tserver with Master: 67926f00bb464bb596118e62f02ddddc (127.16.61.1:42995)
I20260619 11:06:40.071013 27870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:48139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:06:40.095302 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:53789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43325
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 11:06:40.252703 28090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:06:40.253057 28090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:06:40.253121 28090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:06:40.259166 28090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:06:40.259241 28090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:06:40.259326 28090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:06:40.267084 28090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.61.20:53789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:06:40.268368 28090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:06:40.269685 28090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:06:40.284212 28099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:06:40.284212 28097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:06:40.284265 28096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:06:40.284969 28090 server_base.cc:1034] running on GCE node
I20260619 11:06:40.285358 28090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:06:40.286072 28090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:06:40.287321 28090 hybrid_clock.cc:648] HybridClock initialized: now 1781867200287282 us; error 59 us; skew 500 ppm
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:06:40.290830 28090 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:06:40.291972 28090 webserver.cc:460] Webserver started at http://127.16.61.2:46475/ using document root <none> and password file <none>
I20260619 11:06:40.292536 28090 fs_manager.cc:359] Metadata directory not provided
I20260619 11:06:40.292629 28090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:06:40.292863 28090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:06:40.314577 28090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "923f87d1a6354246abc4c09e2e357aec"
format_stamp: "Formatted at 2026-06-19 11:06:40 on dist-test-slave-8vcz"
server_key: "1852bd18db704b348f5783f6eb3dd855"
server_key_iv: "d541dfa6bc6d433f271814348e8d927e"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:40.315409 28090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "923f87d1a6354246abc4c09e2e357aec"
format_stamp: "Formatted at 2026-06-19 11:06:40 on dist-test-slave-8vcz"
server_key: "1852bd18db704b348f5783f6eb3dd855"
server_key_iv: "d541dfa6bc6d433f271814348e8d927e"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:40.324414 28090 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260619 11:06:40.349082 28106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:40.350787 28090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:06:40.350914 28090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "923f87d1a6354246abc4c09e2e357aec"
format_stamp: "Formatted at 2026-06-19 11:06:40 on dist-test-slave-8vcz"
server_key: "1852bd18db704b348f5783f6eb3dd855"
server_key_iv: "d541dfa6bc6d433f271814348e8d927e"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:40.351044 28090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:06:40.393370 28090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:06:40.397174 28090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:06:40.397415 28090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:06:40.398132 28090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:06:40.399160 28090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:06:40.399269 28090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:40.399367 28090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:06:40.399428 28090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:40.411016 28090 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:44903
I20260619 11:06:40.411075 28219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:44903 every 8 connection(s)
I20260619 11:06:40.412243 28090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:06:40.414911 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 28090
I20260619 11:06:40.415038 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:06:40.415339 16628 external_mini_cluster.cc:1428] Setting key 32789732f15a611ea57da9dcc117f27f
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:06:40.430716 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.414324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:54139 (local address 127.16.61.62:43325)
0619 11:06:40.414463 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:40.414469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:40.416215 (+ 1746us) server_negotiation.cc:407] Connection header received
0619 11:06:40.417408 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:40.417412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:40.417469 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:40.417534 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:06:40.419123 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.419958 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.422049 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.422402 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.422481 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.425392 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:06:40.425396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:06:40.425399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:06:40.425424 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:06:40.427209 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:40.428314 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:40.428318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:40.428320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:40.428413 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:40.428885 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:40.428889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:40.428890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:40.429112 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:06:40.429283 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:40.429965 (+ 682us) server_negotiation.cc:299] Negotiation successful
0619 11:06:40.430504 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:06:40.432466 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.414833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43325 (local address 127.16.61.2:54139)
0619 11:06:40.416027 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:40.416101 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 11:06:40.417168 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:06:40.417954 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:06:40.417963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:06:40.418306 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:06:40.418964 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.418983 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.420115 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.420119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.421908 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.421919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.422677 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.422680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.422865 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.423512 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:06:40.423532 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:06:40.425203 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:06:40.427373 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:06:40.427381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:06:40.427400 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:06:40.428196 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:06:40.428547 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:06:40.428551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:06:40.428554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:06:40.428756 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:06:40.429249 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:06:40.429264 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:06:40.429626 (+ 362us) client_negotiation.cc:769] Sending connection context
0619 11:06:40.430880 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0619 11:06:40.431252 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":146,"threads_started":1}
I20260619 11:06:40.434373 28220 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43325
I20260619 11:06:40.434633 28220 heartbeater.cc:464] Registering TS with master...
I20260619 11:06:40.435243 28220 heartbeater.cc:511] Master 127.16.61.62:43325 requested a full tablet report, sending...
I20260619 11:06:40.440004 27870 ts_manager.cc:194] Registered new tserver with Master: 923f87d1a6354246abc4c09e2e357aec (127.16.61.2:44903)
I20260619 11:06:40.440760 27870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:54139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:06:40.455231 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:53789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43325
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:06:40.612712 28225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:06:40.613176 28225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:06:40.613293 28225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:06:40.619426 28225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:06:40.619555 28225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:06:40.619694 28225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:06:40.626995 28225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.61.20:53789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:06:40.628454 28225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:06:40.629776 28225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:06:40.642923 28225 server_base.cc:1034] running on GCE node
W20260619 11:06:40.642815 28231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:06:40.643327 28230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:06:40.643878 28233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:06:40.644361 28225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:06:40.645114 28225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:06:40.646752 28225 hybrid_clock.cc:648] HybridClock initialized: now 1781867200646689 us; error 88 us; skew 500 ppm
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:06:40.650327 28225 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:06:40.651305 28225 webserver.cc:460] Webserver started at http://127.16.61.3:43565/ using document root <none> and password file <none>
I20260619 11:06:40.652009 28225 fs_manager.cc:359] Metadata directory not provided
I20260619 11:06:40.652081 28225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:06:40.652285 28225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:06:40.669360 28225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "2957befa3e4247168ffe69938bc51dea"
format_stamp: "Formatted at 2026-06-19 11:06:40 on dist-test-slave-8vcz"
server_key: "73eae76a6afee3daf8892e8f8a153840"
server_key_iv: "ec3546a064ba85bdf45f5d0d983f0b35"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:40.673743 28225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "2957befa3e4247168ffe69938bc51dea"
format_stamp: "Formatted at 2026-06-19 11:06:40 on dist-test-slave-8vcz"
server_key: "73eae76a6afee3daf8892e8f8a153840"
server_key_iv: "ec3546a064ba85bdf45f5d0d983f0b35"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:40.683770 28225 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.003s
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:06:40.708022 28240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:40.709790 28225 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260619 11:06:40.709968 28225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "2957befa3e4247168ffe69938bc51dea"
format_stamp: "Formatted at 2026-06-19 11:06:40 on dist-test-slave-8vcz"
server_key: "73eae76a6afee3daf8892e8f8a153840"
server_key_iv: "ec3546a064ba85bdf45f5d0d983f0b35"
server_key_version: "kuduclusterkey@0"
I20260619 11:06:40.714730 28225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:06:40.733851 28225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:06:40.737221 28225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:06:40.737455 28225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:06:40.738142 28225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:06:40.739128 28225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:06:40.739224 28225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:40.739331 28225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:06:40.739390 28225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:06:40.750002 28225 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:40357
I20260619 11:06:40.751081 28225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:06:40.756198 28353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:40357 every 8 connection(s)
I20260619 11:06:40.760730 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 28225
I20260619 11:06:40.760848 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:06:40.761132 16628 external_mini_cluster.cc:1428] Setting key 59c0cd4040d4c9f0d2a304a5a03f126a
Jun 19 11:06:40 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:06:40.778600 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.758502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:51275 (local address 127.16.61.62:43325)
0619 11:06:40.758635 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:40.758640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:40.763759 (+ 5119us) server_negotiation.cc:407] Connection header received
0619 11:06:40.764883 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:40.764887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:40.764941 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:40.765032 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:06:40.767338 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.768124 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.769822 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.770126 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.770195 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.774379 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:06:40.774383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:06:40.774386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:06:40.774414 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:06:40.776193 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:40.776823 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:40.776827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:40.776828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:40.776915 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:40.777330 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:40.777333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:40.777335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:40.777552 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:06:40.777623 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:40.778324 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 11:06:40.778443 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:06:40.780120 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.759095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43325 (local address 127.16.61.3:51275)
0619 11:06:40.763596 (+ 4501us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:40.763653 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 11:06:40.764654 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:06:40.766209 (+ 1555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:06:40.766218 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:06:40.766558 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:06:40.767191 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.767209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.768303 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.768307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.769669 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.769680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.770387 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.770390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.770575 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.772517 (+ 1942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:06:40.772538 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:06:40.774158 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:06:40.776347 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:06:40.776353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:06:40.776370 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:06:40.776698 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:06:40.777028 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:06:40.777031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:06:40.777033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:06:40.777202 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:06:40.777706 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:06:40.777712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:06:40.778027 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 11:06:40.778783 (+ 756us) client_negotiation.cc:240] Negotiation successful
0619 11:06:40.779102 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4258,"thread_start_us":147,"threads_started":1}
I20260619 11:06:40.781490 28354 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43325
I20260619 11:06:40.781715 28354 heartbeater.cc:464] Registering TS with master...
I20260619 11:06:40.782253 28354 heartbeater.cc:511] Master 127.16.61.62:43325 requested a full tablet report, sending...
I20260619 11:06:40.783334 27870 ts_manager.cc:194] Registered new tserver with Master: 2957befa3e4247168ffe69938bc51dea (127.16.61.3:40357)
I20260619 11:06:40.784159 27870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:51275
I20260619 11:06:40.786305 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:06:40.799011 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.789387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.16.61.62:43325)
0619 11:06:40.789512 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:40.789516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:40.789833 (+ 317us) server_negotiation.cc:407] Connection header received
0619 11:06:40.790004 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:40.790007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:40.790050 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:40.790096 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:06:40.791094 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.791919 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.793118 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.793442 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.793510 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.795223 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:06:40.795227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:06:40.795230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:06:40.795253 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:06:40.797064 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:40.797719 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:40.797722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:40.797724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:40.797814 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:06:40.798098 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:06:40.798102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:06:40.798103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:06:40.798292 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:06:40.798391 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:40.798728 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 11:06:40.798850 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:06:40.803211 27870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:06:40.807905 27870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:06:40.856549 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.843484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42995 (local address 127.0.0.1:47638)
0619 11:06:40.848560 (+ 5076us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:40.848574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:06:40.848666 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:06:40.849444 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:06:40.849453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:06:40.849477 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:06:40.849604 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.849611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.852088 (+ 2477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.852091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.853647 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.853657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.856005 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.856010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.856122 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.856164 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 11:06:40.856275 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:06:40.856398 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4335,"mutex_wait_us":187,"thread_start_us":81,"threads_started":1}
I20260619 11:06:40.857219 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.843640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47638 (local address 127.16.61.1:42995)
0619 11:06:40.847729 (+ 4089us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:40.847737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:40.848763 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 11:06:40.848859 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:40.848865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:40.849074 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:40.849215 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:06:40.849739 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.850946 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.853810 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.854692 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.854809 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.854962 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:40.856925 (+ 1963us) server_negotiation.cc:299] Negotiation successful
0619 11:06:40.857084 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3961,"thread_start_us":103,"threads_started":1}
I20260619 11:06:40.859094 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.844317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.16.61.3:40357)
0619 11:06:40.852798 (+ 8481us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:40.852809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:40.852831 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:06:40.852902 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:40.852906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:40.853084 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:40.853224 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:06:40.853601 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.855510 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.857183 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.858063 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.858170 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.858295 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:40.858865 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 11:06:40.858960 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8364,"thread_start_us":125,"threads_started":1}
I20260619 11:06:40.862793 28010 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9981e22af24d66b5b113c5b5404cdc (DEFAULT_TABLE table=test-table [id=8ef1459057f84a40a5fbf2cdce61efb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:06:40.863967 28010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9981e22af24d66b5b113c5b5404cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:06:40.858758 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.843872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40357 (local address 127.0.0.1:54030)
0619 11:06:40.851905 (+ 8033us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:40.851918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:06:40.851997 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:06:40.853359 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:06:40.853362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:06:40.853379 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:06:40.853472 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.853477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.855634 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.855637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.856976 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.856987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.858356 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.858359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.858461 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.858486 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:06:40.858557 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:06:40.858622 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7972,"thread_start_us":87,"threads_started":1}
I20260619 11:06:40.871447 28273 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9981e22af24d66b5b113c5b5404cdc (DEFAULT_TABLE table=test-table [id=8ef1459057f84a40a5fbf2cdce61efb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:06:40.872541 28273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9981e22af24d66b5b113c5b5404cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:06:40.873507 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.842628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:44903 (local address 127.0.0.1:59152)
0619 11:06:40.844274 (+ 1646us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:40.844325 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:06:40.844448 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:06:40.852510 (+ 8062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:06:40.852513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:06:40.852530 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:06:40.852664 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.852670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.858143 (+ 5473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.858147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.859516 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:40.859527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.873078 (+ 13551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.873081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:40.873181 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.873206 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:06:40.873285 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:06:40.873364 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1473,"mutex_wait_us":2,"thread_start_us":112,"threads_started":1}
I20260619 11:06:40.874184 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:40.843880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.16.61.2:44903)
0619 11:06:40.851884 (+ 8004us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:40.851898 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:40.851924 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:06:40.851999 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:40.852005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:40.852200 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:40.852365 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:06:40.855035 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.858011 (+ 2976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.868393 (+ 10382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:40.869238 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:40.869369 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:40.869500 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:40.873929 (+ 4429us) server_negotiation.cc:299] Negotiation successful
0619 11:06:40.874049 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7895,"thread_start_us":134,"threads_started":1}
I20260619 11:06:40.877161 28145 tablet_service.cc:1467] Processing CreateTablet for tablet 3c9981e22af24d66b5b113c5b5404cdc (DEFAULT_TABLE table=test-table [id=8ef1459057f84a40a5fbf2cdce61efb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:06:40.878196 28145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c9981e22af24d66b5b113c5b5404cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:06:40.881353 28373 tablet_bootstrap.cc:492] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Bootstrap starting.
I20260619 11:06:40.884253 28373 tablet_bootstrap.cc:654] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Neither blocks nor log segments found. Creating new log.
I20260619 11:06:40.885169 28373 log.cc:824] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Log is configured to *not* fsync() on all Append() calls
I20260619 11:06:40.887240 28373 tablet_bootstrap.cc:492] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: No bootstrap required, opened a new log
I20260619 11:06:40.887336 28372 tablet_bootstrap.cc:492] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc: Bootstrap starting.
I20260619 11:06:40.887420 28373 ts_tablet_manager.cc:1397] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:06:40.889564 28372 tablet_bootstrap.cc:654] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc: Neither blocks nor log segments found. Creating new log.
I20260619 11:06:40.891199 28372 log.cc:824] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc: Log is configured to *not* fsync() on all Append() calls
I20260619 11:06:40.892345 28373 raft_consensus.cc:348] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:40.892692 28373 raft_consensus.cc:374] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:06:40.892795 28373 raft_consensus.cc:729] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2957befa3e4247168ffe69938bc51dea, State: Initialized, Role: FOLLOWER
I20260619 11:06:40.893126 28372 tablet_bootstrap.cc:492] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc: No bootstrap required, opened a new log
I20260619 11:06:40.893321 28372 ts_tablet_manager.cc:1397] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 11:06:40.893440 28373 consensus_queue.cc:260] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:40.894492 28373 ts_tablet_manager.cc:1428] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 11:06:40.894893 28354 heartbeater.cc:503] Master 127.16.61.62:43325 was elected leader, sending a full tablet report...
I20260619 11:06:40.897645 28372 raft_consensus.cc:348] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:40.897887 28372 raft_consensus.cc:374] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:06:40.897931 28372 raft_consensus.cc:729] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67926f00bb464bb596118e62f02ddddc, State: Initialized, Role: FOLLOWER
I20260619 11:06:40.898360 28372 consensus_queue.cc:260] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:40.899192 28375 tablet_bootstrap.cc:492] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Bootstrap starting.
I20260619 11:06:40.899266 28372 ts_tablet_manager.cc:1428] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260619 11:06:40.899603 28082 heartbeater.cc:503] Master 127.16.61.62:43325 was elected leader, sending a full tablet report...
I20260619 11:06:40.902843 28375 tablet_bootstrap.cc:654] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Neither blocks nor log segments found. Creating new log.
I20260619 11:06:40.904209 28375 log.cc:824] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Log is configured to *not* fsync() on all Append() calls
I20260619 11:06:40.906527 28375 tablet_bootstrap.cc:492] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: No bootstrap required, opened a new log
I20260619 11:06:40.906730 28375 ts_tablet_manager.cc:1397] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 11:06:40.911203 28375 raft_consensus.cc:348] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:40.911485 28375 raft_consensus.cc:374] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:06:40.911593 28375 raft_consensus.cc:729] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923f87d1a6354246abc4c09e2e357aec, State: Initialized, Role: FOLLOWER
I20260619 11:06:40.912109 28375 consensus_queue.cc:260] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:40.912798 28220 heartbeater.cc:503] Master 127.16.61.62:43325 was elected leader, sending a full tablet report...
I20260619 11:06:40.913156 28375 ts_tablet_manager.cc:1428] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260619 11:06:40.913484 28221 tablet.cc:2369] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:06:41.008003 28355 tablet.cc:2369] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 11:06:41 dist-test-slave-8vcz krb5kdc[27394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867200, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
W20260619 11:06:41.049288 28083 tablet.cc:2369] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:06:41.050007 28380 raft_consensus.cc:482] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:06:41.050215 28380 raft_consensus.cc:504] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:41.051622 28380 leader_election.cc:288] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2957befa3e4247168ffe69938bc51dea (127.16.61.3:40357), 67926f00bb464bb596118e62f02ddddc (127.16.61.1:42995)
I20260619 11:06:41.057580 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:41.052897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:34715 (local address 127.16.61.3:40357)
0619 11:06:41.053024 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:41.053031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:41.053050 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:06:41.053108 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:41.053110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:41.053158 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:41.053242 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:06:41.053904 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.054662 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.056252 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.056863 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.056938 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:41.057013 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:41.057361 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 11:06:41.057451 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:06:41.057698 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:41.052156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:42661 (local address 127.16.61.1:42995)
0619 11:06:41.052263 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:41.052267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:41.053771 (+ 1504us) server_negotiation.cc:407] Connection header received
0619 11:06:41.053946 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:41.053948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:41.053992 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:41.054037 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:06:41.054386 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.055088 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.056505 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.057059 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.057132 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:41.057184 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:41.057516 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 11:06:41.057582 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 11:06:41.058142 28383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:41.052089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42995 (local address 127.16.61.2:42661)
0619 11:06:41.053686 (+ 1597us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:41.053700 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:06:41.053832 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:06:41.054162 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:06:41.054164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:06:41.054180 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:06:41.054275 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:41.054280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.055220 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.055223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:41.056336 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:41.056344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.057260 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.057263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:41.057362 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:41.057382 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:06:41.057920 (+ 538us) client_negotiation.cc:240] Negotiation successful
0619 11:06:41.057992 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":110,"threads_started":1}
I20260619 11:06:41.058260 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:41.051807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40357 (local address 127.16.61.2:34715)
0619 11:06:41.052446 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:41.052461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:06:41.052575 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:06:41.053596 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:06:41.053602 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:06:41.053628 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:06:41.053766 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:41.053773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.054811 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.054814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:41.056110 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:06:41.056120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.057133 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.057136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:06:41.057226 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:41.057251 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:06:41.058034 (+ 783us) client_negotiation.cc:240] Negotiation successful
0619 11:06:41.058089 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":99,"threads_started":1}
I20260619 11:06:41.058908 28018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9981e22af24d66b5b113c5b5404cdc" candidate_uuid: "923f87d1a6354246abc4c09e2e357aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67926f00bb464bb596118e62f02ddddc" is_pre_election: true
I20260619 11:06:41.059585 28018 raft_consensus.cc:2454] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923f87d1a6354246abc4c09e2e357aec in term 0.
I20260619 11:06:41.060279 28110 leader_election.cc:302] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67926f00bb464bb596118e62f02ddddc, 923f87d1a6354246abc4c09e2e357aec; no voters:
I20260619 11:06:41.060264 28305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9981e22af24d66b5b113c5b5404cdc" candidate_uuid: "923f87d1a6354246abc4c09e2e357aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2957befa3e4247168ffe69938bc51dea" is_pre_election: true
I20260619 11:06:41.060554 28305 raft_consensus.cc:2454] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923f87d1a6354246abc4c09e2e357aec in term 0.
I20260619 11:06:41.060565 28380 raft_consensus.cc:2781] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:06:41.060665 28380 raft_consensus.cc:482] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:06:41.060716 28380 raft_consensus.cc:3037] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:06:41.061944 28380 raft_consensus.cc:504] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:41.062268 28380 leader_election.cc:288] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [CANDIDATE]: Term 1 election: Requested vote from peers 2957befa3e4247168ffe69938bc51dea (127.16.61.3:40357), 67926f00bb464bb596118e62f02ddddc (127.16.61.1:42995)
I20260619 11:06:41.062737 28018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9981e22af24d66b5b113c5b5404cdc" candidate_uuid: "923f87d1a6354246abc4c09e2e357aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67926f00bb464bb596118e62f02ddddc"
I20260619 11:06:41.062862 28018 raft_consensus.cc:3037] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:06:41.063385 28305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c9981e22af24d66b5b113c5b5404cdc" candidate_uuid: "923f87d1a6354246abc4c09e2e357aec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2957befa3e4247168ffe69938bc51dea"
I20260619 11:06:41.063501 28305 raft_consensus.cc:3037] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:06:41.064066 28018 raft_consensus.cc:2454] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923f87d1a6354246abc4c09e2e357aec in term 1.
I20260619 11:06:41.064376 28110 leader_election.cc:302] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67926f00bb464bb596118e62f02ddddc, 923f87d1a6354246abc4c09e2e357aec; no voters:
I20260619 11:06:41.064564 28380 raft_consensus.cc:2781] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:06:41.064744 28380 raft_consensus.cc:686] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 1 LEADER]: Becoming Leader. State: Replica: 923f87d1a6354246abc4c09e2e357aec, State: Running, Role: LEADER
I20260619 11:06:41.064869 28305 raft_consensus.cc:2454] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923f87d1a6354246abc4c09e2e357aec in term 1.
I20260619 11:06:41.065091 28380 consensus_queue.cc:237] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } }
I20260619 11:06:41.068457 27870 catalog_manager.cc:5495] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec reported cstate change: term changed from 0 to 1, leader changed from <none> to 923f87d1a6354246abc4c09e2e357aec (127.16.61.2). New cstate: current_term: 1 leader_uuid: "923f87d1a6354246abc4c09e2e357aec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923f87d1a6354246abc4c09e2e357aec" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 } health_report { overall_health: UNKNOWN } } }
I20260619 11:06:41.085621 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:41.080276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.16.61.2:44903)
0619 11:06:41.080383 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:06:41.080387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:06:41.080495 (+ 108us) server_negotiation.cc:407] Connection header received
0619 11:06:41.080678 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:06:41.080681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:06:41.080747 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:06:41.080858 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:06:41.081222 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.082023 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.083471 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:06:41.083851 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:06:41.083950 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:06:41.084356 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:06:41.084892 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:06:41.085003 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:06:41.085426 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 11:06:41.085483 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 11:06:41.092514 28018 raft_consensus.cc:1261] T 3c9981e22af24d66b5b113c5b5404cdc P 67926f00bb464bb596118e62f02ddddc [term 1 FOLLOWER]: Refusing update from remote peer 923f87d1a6354246abc4c09e2e357aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:06:41.092650 28305 raft_consensus.cc:1261] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 1 FOLLOWER]: Refusing update from remote peer 923f87d1a6354246abc4c09e2e357aec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:06:41.093281 28380 consensus_queue.cc:1035] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2957befa3e4247168ffe69938bc51dea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:06:41.093488 28385 consensus_queue.cc:1035] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [LEADER]: Connected to new peer: Peer: permanent_uuid: "67926f00bb464bb596118e62f02ddddc" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 42995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:06:41.106153 28389 mvcc.cc:205] Tried to move back new op lower bound from 7298528055667453952 to 7298528055565172736. Current Snapshot: MvccSnapshot[applied={T|T < 7298528055667453952}]
I20260619 11:06:41.114548 28390 mvcc.cc:205] Tried to move back new op lower bound from 7298528055667453952 to 7298528055565172736. Current Snapshot: MvccSnapshot[applied={T|T < 7298528055667453952}]
I20260619 11:06:41.125367 27869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:06:41.125540 27869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:06:41.128340 27869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788d3445b4e3446ead9e15aac2efb02a: Sending DeleteTablet for 3 replicas of tablet 3c9981e22af24d66b5b113c5b5404cdc
I20260619 11:06:41.129540 28010 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c9981e22af24d66b5b113c5b5404cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:06:41 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:47638
I20260619 11:06:41.130277 28273 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c9981e22af24d66b5b113c5b5404cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:06:41 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:54030
I20260619 11:06:41.131299 28145 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c9981e22af24d66b5b113c5b5404cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:06:41 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:59152
I20260619 11:06:41.132323 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 27951
I20260619 11:06:41.139869 28400 tablet_replica.cc:330] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: stopping tablet replica
I20260619 11:06:41.140162 28399 tablet_replica.cc:330] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: stopping tablet replica
I20260619 11:06:41.140615 28399 raft_consensus.cc:2229] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:06:41.140985 28400 raft_consensus.cc:2229] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:06:41.141485 28400 raft_consensus.cc:2258] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:06:41.141865 28399 raft_consensus.cc:2258] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:06:41.143657 28400 ts_tablet_manager.cc:1905] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:06:41.146564 28400 ts_tablet_manager.cc:1918] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:06:41.148797 28399 ts_tablet_manager.cc:1905] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:06:41.151389 28399 ts_tablet_manager.cc:1918] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:06:41.156494 28400 log.cc:1196] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/3c9981e22af24d66b5b113c5b5404cdc
W20260619 11:06:41.156780 27857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:42995 (error 108)
I20260619 11:06:41.157887 28400 ts_tablet_manager.cc:1939] T 3c9981e22af24d66b5b113c5b5404cdc P 923f87d1a6354246abc4c09e2e357aec: Deleting consensus metadata
I20260619 11:06:41.158141 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 28090
I20260619 11:06:41.158815 28399 log.cc:1196] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/3c9981e22af24d66b5b113c5b5404cdc
I20260619 11:06:41.159158 28399 ts_tablet_manager.cc:1939] T 3c9981e22af24d66b5b113c5b5404cdc P 2957befa3e4247168ffe69938bc51dea: Deleting consensus metadata
I20260619 11:06:41.160531 27856 catalog_manager.cc:4841] TS 2957befa3e4247168ffe69938bc51dea (127.16.61.3:40357): tablet 3c9981e22af24d66b5b113c5b5404cdc (table test-table [id=8ef1459057f84a40a5fbf2cdce61efb6]) successfully deleted
I20260619 11:06:41.167802 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 28225
I20260619 11:06:41.172573 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:41.172274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:42995 (local address 127.0.0.1:47648)
0619 11:06:41.172405 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:41.172483 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 11:06:41.172930 27857 catalog_manager.cc:4584] TS 67926f00bb464bb596118e62f02ddddc (127.16.61.1:42995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c9981e22af24d66b5b113c5b5404cdc: Network error: Client connection negotiation failed: client connection to 127.16.61.1:42995: connect: Connection refused (error 111)
I20260619 11:06:41.173399 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:06:41.172644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:44903 (local address 127.0.0.1:59172)
0619 11:06:41.173276 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 11:06:41.173324 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.2:44903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":132}
I20260619 11:06:41.175160 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 27838
I20260619 11:06:41.181555 16628 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 11:06:41.525452 16628 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 11:06:41.525575 16628 mini_ranger.cc:66] Stopping Ranger...
I20260619 11:06:41.869552 16628 mini_ranger.cc:68] Stopped Ranger
2026-06-19 11:06:41.869 UTC [27424] LOG: received smart shutdown request
2026-06-19 11:06:41.873 UTC [27424] LOG: background worker "logical replication launcher" (PID 27433) exited with exit code 1
2026-06-19 11:06:41.873 UTC [27428] LOG: shutting down
2026-06-19 11:06:42.020 UTC [27424] LOG: database system is shut down
2026-06-19T11:06:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45378 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:06:42 dist-test-slave-8vcz krb5kdc[28410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:06:42 dist-test-slave-8vcz krb5kdc[28410](info): set up 2 sockets
Jun 19 11:06:42 dist-test-slave-8vcz krb5kdc[28410](info): commencing operation
krb5kdc: starting...
W20260619 11:06:43.159601 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:06:43 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:06:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/rangeradmin_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/rangeradmin_127.16.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/rangerlookup_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/rangerlookup_127.16.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/HTTP_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/HTTP_127.16.61.20@KRBTEST.COM.keytab.
I20260619 11:06:43.312184 16628 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIllhcy/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/postgres -l logfile start
2026-06-19 11:06:45.653 UTC [28440] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 11:06:45.653 UTC [28440] LOG: listening on IPv4 address "127.16.61.20", port 50841
2026-06-19 11:06:45.657 UTC [28440] LOG: listening on Unix socket "/tmp/.s.PGSQL.50841"
2026-06-19 11:06:45.678 UTC [28443] LOG: database system was shut down at 2026-06-19 11:06:44 UTC
2026-06-19 11:06:45.681 UTC [28440] LOG: database system is ready to accept connections
I20260619 11:06:46.635478 16628 mini_postgres.cc:96] Postgres bound to 50841
2026-06-19 11:06:46.641 UTC [28451] FATAL: database "slave" does not exist
127.16.61.20:50841 - accepting connections
I20260619 11:06:46.642067 16628 mini_ranger.cc:161] Starting Ranger...
I20260619 11:06:46.656284 16628 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 11:06:47.089325 16628 mini_ranger.cc:87] Created ranger Postgres database
I20260619 11:06:47.089411 16628 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-admin
2026-06-19 11:06:47,357 [I] DB FLAVOR :POSTGRES
2026-06-19 11:06:47,358 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:06:47,358 [I] Checking connection..
2026-06-19 11:06:47,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 11:06:47,587 [I] Checking connection passed.
2026-06-19 11:06:47,588 [I] --------- Verifying version history table ---------
2026-06-19 11:06:47,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:06:47,843 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:06:47,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:06:48,093 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 11:06:48,093 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 11:06:48,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 11:06:48.315 UTC [28515] WARNING: there is no transaction in progress
2026-06-19 11:06:48,321 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 11:06:48,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 11:06:48,542 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 11:06:48,542 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:06:48,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 11:06:48,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 11:06:49,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8vcz.c.gcp-upstream.internal','N') ;"
2026-06-19 11:06:49,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:06:49,452 [I] Table x_portal_user does not exist in database ranger
2026-06-19 11:06:49,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:06:49,678 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 11:06:49,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:06:49,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:06:50,133 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 11:06:50,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 11:06:51.017 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.032 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.047 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.062 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.078 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.096 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.158 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.164 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.178 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.188 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.198 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.211 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.221 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.228 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.235 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51.242 UTC [28623] WARNING: there is no transaction in progress
2026-06-19 11:06:51,841 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 11:06:51,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 11:06:52,070 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 11:06:52,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 11:06:52,293 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 11:06:52,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 11:06:52,534 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 11:06:52,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 11:06:52,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 11:06:53,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8vcz.c.gcp-upstream.internal';"
2026-06-19 11:06:53,213 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 11:06:53,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 11:06:53,457 [I] DB_PATCHES have already been applied
I20260619 11:06:53.463109 16628 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIllhcy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 11:06:53.463172 16628 mini_ranger.cc:193] Using host: 127.16.61.20
I20260619 11:06:53.466427 16628 mini_ranger.cc:227] Ranger admin URL: http://127.16.61.20:58559
Jun 19, 2026 11:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 11:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 11:06:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 11:06:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 11:06:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 11:06:54.580863 16628 mini_ranger.cc:160] Time spent starting Ranger: real 7.939s user 0.002s sys 0.007s
Jun 19, 2026 11:07:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 11:07:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 11:07:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 11:07:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 11:07:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 11:07:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 11:07:15.354882 16628 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.61.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.61.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/rangerkms_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.61.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/rangerkms_127.16.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.61.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.61.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.61.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/HTTP_127.16.61.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.61.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/HTTP_127.16.61.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 11:07:15 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 11:07:15.432413 16628 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 11:07:15.441303 16628 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 11:07:16.224767 16628 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 11:07:16.224831 16628 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms
I20260619 11:07:16.224845 16628 mini_ranger_kms.cc:227] Using postgres at 127.16.61.20:50841
2026-06-19 11:07:16,393 [I] DB FLAVOR :POSTGRES
2026-06-19 11:07:16,393 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 11:07:16,394 [I] Checking connection
2026-06-19 11:07:16,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 11:07:16,626 [I] connection success
2026-06-19 11:07:16,627 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 11:07:16,627 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 11:07:16,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 11:07:16,866 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 11:07:16,866 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 11:07:16,866 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 11:07:16,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.61.20:50841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 11:07:17,122 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 11:07:17.549233 16628 mini_ranger_kms.cc:310] Created kms service
I20260619 11:07:17.718532 16628 mini_ranger_kms.cc:326] Created kudu user
I20260619 11:07:17.759060 16628 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 11:07:17 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:18.109712 16628 mini_ranger_kms.cc:384] Added ranger policy
I20260619 11:07:18.109861 16628 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIllhcy/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIllhcy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 11:07:18.109881 16628 mini_ranger_kms.cc:240] Using host: 127.16.61.20
I20260619 11:07:18.114025 16628 mini_ranger_kms.cc:276] Ranger KMS PID: 28803
I20260619 11:07:18.114101 16628 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.61.20:50017
Jun 19, 2026 11:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 11:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 11:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 11:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 11:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 11:07:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 11:07:20.297511 16628 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.865s user 0.007s sys 0.142s
I20260619 11:07:20.297652 16628 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 11:07:20.297767 16628 mini_ranger_kms.cc:401] 127.16.61.20:50017/kms/v1/keys
Jun 19 11:07:20 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.61.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 11:07:20 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.61.20@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
Jun 19 11:07:21 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.61.20@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
Jun 19 11:07:23 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:07:23.457060 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:46033
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:46033
--ranger_config_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:50017/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:23.564203 28854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:23.564440 28854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:23.564478 28854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:23.568416 28854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:23.568467 28854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:23.568488 28854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:23.568531 28854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:23.568562 28854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:23.573855 28854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.61.20:50017/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:46033
--ranger_config_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:46033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:23.574810 28854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:23.575783 28854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:23.584401 28860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:23.584548 28854 server_base.cc:1034] running on GCE node
W20260619 11:07:23.584416 28859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:23.584389 28862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:23.585178 28854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:23.586241 28854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:23.587422 28854 hybrid_clock.cc:648] HybridClock initialized: now 1781867243587408 us; error 30 us; skew 500 ppm
Jun 19 11:07:23 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867243, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:23.590204 28854 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:07:23.591131 28854 webserver.cc:460] Webserver started at http://127.16.61.62:41823/ using document root <none> and password file <none>
I20260619 11:07:23.591804 28854 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:23.591874 28854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:23.592082 28854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:07:23 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867243, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:23.650309 28854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "b9ea57336f824299bb98dc5a79a04128"
format_stamp: "Formatted at 2026-06-19 11:07:23 on dist-test-slave-8vcz"
server_key: "49d25c28d0f0c859c0feb663c6ec1634"
server_key_iv: "eba6d9ea6eab1941b696982eab9350c7"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:23.651027 28854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "b9ea57336f824299bb98dc5a79a04128"
format_stamp: "Formatted at 2026-06-19 11:07:23 on dist-test-slave-8vcz"
server_key: "49d25c28d0f0c859c0feb663c6ec1634"
server_key_iv: "eba6d9ea6eab1941b696982eab9350c7"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:23.655814 28854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 11:07:23.672161 28869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:23.673837 28854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:07:23.673965 28854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "b9ea57336f824299bb98dc5a79a04128"
format_stamp: "Formatted at 2026-06-19 11:07:23 on dist-test-slave-8vcz"
server_key: "49d25c28d0f0c859c0feb663c6ec1634"
server_key_iv: "eba6d9ea6eab1941b696982eab9350c7"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:23.674113 28854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:23.690615 28854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:23.693698 28854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:23.693877 28854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:23.701987 28921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:46033 every 8 connection(s)
I20260619 11:07:23.702004 28854 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:46033
I20260619 11:07:23.703207 28854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:07:23.703558 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 28854
I20260619 11:07:23.703706 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:07:23.704058 16628 external_mini_cluster.cc:1428] Setting key 63f87602fadae273ead49c49ecc63c1e
I20260619 11:07:23.706548 28922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:23.712725 28922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Bootstrap starting.
Jun 19 11:07:23 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:23.715371 28922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:23.716282 28922 log.cc:824] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:23.718322 28922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: No bootstrap required, opened a new log
I20260619 11:07:23.720614 28925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:23.705544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.16.61.62:46033)
0619 11:07:23.705986 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:23.706004 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:23.706047 (+ 43us) server_negotiation.cc:407] Connection header received
0619 11:07:23.706899 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:23.706922 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:23.707350 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:23.707723 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:23.708604 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:23.710151 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:23.711343 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:23.712067 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:23.712213 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:23.715293 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:23.715300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:23.715315 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:23.715345 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:23.717846 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:23.718295 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:23.718307 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:23.718312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:23.718504 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:23.718793 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:23.718796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:23.718798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:23.719164 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:23.719379 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:23.719633 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 11:07:23.719905 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":131,"threads_started":1}
I20260619 11:07:23.722083 28922 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } }
I20260619 11:07:23.722332 28922 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:23.722378 28922 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ea57336f824299bb98dc5a79a04128, State: Initialized, Role: FOLLOWER
I20260619 11:07:23.722867 28922 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } }
I20260619 11:07:23.723023 28922 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:07:23.723104 28922 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:07:23.723198 28922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:23.724268 28922 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } }
I20260619 11:07:23.724666 28922 leader_election.cc:302] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ea57336f824299bb98dc5a79a04128; no voters:
I20260619 11:07:23.724961 28922 leader_election.cc:288] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:07:23.725069 28927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:23.725320 28927 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 1 LEADER]: Becoming Leader. State: Replica: b9ea57336f824299bb98dc5a79a04128, State: Running, Role: LEADER
I20260619 11:07:23.726102 28922 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:07:23.726248 28927 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } }
I20260619 11:07:23.727883 28929 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ea57336f824299bb98dc5a79a04128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } } }
I20260619 11:07:23.728003 28929 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:23.728152 28922 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:07:23.728211 28929 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ea57336f824299bb98dc5a79a04128. Latest consensus state: current_term: 1 leader_uuid: "b9ea57336f824299bb98dc5a79a04128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } } }
I20260619 11:07:23.728278 28929 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:23.732249 28941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:07:23.736145 28941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:07:23.746063 28941 catalog_manager.cc:1347] Generated new cluster ID: be7a93855f354eaaa734e5b1dbb00207
I20260619 11:07:23.746129 28941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:07:23.758893 28941 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:07:23.760869 28941 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:07:23.767892 28941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Generated new TSK 0
I20260619 11:07:23.768695 28941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:07:24.451112 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:50017/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46033
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:24.564386 28967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:24.564612 28967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:24.564652 28967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:24.568759 28967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:24.568813 28967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:24.568869 28967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:07:24.574080 28967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.61.20:50017/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:24.575141 28967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:24.576301 28967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:24.585821 28972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:24.585875 28973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:24.586862 28975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:24.587136 28967 server_base.cc:1034] running on GCE node
I20260619 11:07:24.587502 28967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:24.588155 28967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:24.589378 28967 hybrid_clock.cc:648] HybridClock initialized: now 1781867244589358 us; error 35 us; skew 500 ppm
Jun 19 11:07:24 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867244, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:24.592355 28967 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:07:24.593443 28967 webserver.cc:460] Webserver started at http://127.16.61.1:45743/ using document root <none> and password file <none>
I20260619 11:07:24.594107 28967 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:24.594190 28967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:24.594417 28967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:07:24 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867244, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:24.609365 28967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "a2c10d5dc19543f9b1c5a2cd16d79251"
format_stamp: "Formatted at 2026-06-19 11:07:24 on dist-test-slave-8vcz"
server_key: "833279b4712712a138abfd69e7dd2407"
server_key_iv: "ff4a52691b77d18e077b84ff127d6cbf"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:24.609972 28967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "a2c10d5dc19543f9b1c5a2cd16d79251"
format_stamp: "Formatted at 2026-06-19 11:07:24 on dist-test-slave-8vcz"
server_key: "833279b4712712a138abfd69e7dd2407"
server_key_iv: "ff4a52691b77d18e077b84ff127d6cbf"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:24.614327 28967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 11:07:24.629465 28984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:24.631068 28967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 11:07:24.631187 28967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "a2c10d5dc19543f9b1c5a2cd16d79251"
format_stamp: "Formatted at 2026-06-19 11:07:24 on dist-test-slave-8vcz"
server_key: "833279b4712712a138abfd69e7dd2407"
server_key_iv: "ff4a52691b77d18e077b84ff127d6cbf"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:24.631304 28967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:24.649768 28967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:24.653223 28967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:24.653445 28967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:24.654132 28967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:24.655113 28967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:24.655215 28967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:24.655332 28967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:24.655407 28967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:24.666273 28967 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:46007
I20260619 11:07:24.666453 29097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:46007 every 8 connection(s)
I20260619 11:07:24.667249 28967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:07:24.667883 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 28967
I20260619 11:07:24.668007 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:07:24.668309 16628 external_mini_cluster.cc:1428] Setting key a918539e5b0d388b1281d743cdf70e2d
Jun 19 11:07:24 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867244, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:24.681887 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:24.669460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46033 (local address 127.16.61.1:53379)
0619 11:07:24.670207 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:24.670256 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:24.671021 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:24.671892 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:24.671901 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:24.672169 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:24.672616 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:24.672629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:24.673460 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:24.673464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:24.674313 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:24.674322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:24.674821 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:24.674825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:24.674996 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:24.675576 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:24.675591 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:24.676949 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:24.678875 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:24.678881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:24.678894 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:24.679148 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:24.679434 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:24.679439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:24.679441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:24.679642 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:24.680151 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:24.680157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:24.680458 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 11:07:24.680649 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 11:07:24.680939 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:07:24.683633 29101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:24.670034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:53379 (local address 127.16.61.62:46033)
0619 11:07:24.671424 (+ 1390us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:24.671433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:24.671446 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:07:24.671483 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:24.671497 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:24.671543 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:24.672729 (+ 1186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:24.672809 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:24.673335 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:24.674435 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:24.674679 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:24.674790 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:24.677116 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:24.677119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:24.677121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:24.677142 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:24.678753 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:24.679249 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:24.679256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:24.679260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:24.679344 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:24.679754 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:24.679758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:24.679760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:24.680008 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:24.680100 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:24.683320 (+ 3220us) server_negotiation.cc:299] Negotiation successful
0619 11:07:24.683452 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":107,"threads_started":1}
I20260619 11:07:24.684747 29098 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46033
I20260619 11:07:24.684998 29098 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:24.685547 29098 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
I20260619 11:07:24.687244 28884 ts_manager.cc:194] Registered new tserver with Master: a2c10d5dc19543f9b1c5a2cd16d79251 (127.16.61.1:46007)
I20260619 11:07:24.688747 28884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:53379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:07:24.719626 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:50017/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46033
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:07:24 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867244, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 11:07:24.853353 29104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:24.853609 29104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:24.853708 29104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:24.857810 29104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:24.857887 29104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:24.857998 29104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:07:24.863216 29104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.61.20:50017/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:24.864259 29104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:24.865233 29104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:24.874296 29112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:24.874346 29114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:24.874306 29111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:24.874701 29104 server_base.cc:1034] running on GCE node
I20260619 11:07:24.875066 29104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:24.875631 29104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:24.876861 29104 hybrid_clock.cc:648] HybridClock initialized: now 1781867244876842 us; error 36 us; skew 500 ppm
Jun 19 11:07:24 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867244, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:24.880050 29104 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:07:24.881062 29104 webserver.cc:460] Webserver started at http://127.16.61.2:35277/ using document root <none> and password file <none>
I20260619 11:07:24.881742 29104 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:24.881834 29104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:24.882048 29104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:07:24 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867244, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:24.897212 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "18e4dd09dbab44b39bfdfda9ca3717ee"
format_stamp: "Formatted at 2026-06-19 11:07:24 on dist-test-slave-8vcz"
server_key: "b67cd284c90d75d1be44d5aa315e8779"
server_key_iv: "c1ae153d41106e44357352f4827624e6"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:24.897876 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "18e4dd09dbab44b39bfdfda9ca3717ee"
format_stamp: "Formatted at 2026-06-19 11:07:24 on dist-test-slave-8vcz"
server_key: "b67cd284c90d75d1be44d5aa315e8779"
server_key_iv: "c1ae153d41106e44357352f4827624e6"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:24.912235 29104 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.006s sys 0.000s
I20260619 11:07:24.937184 29122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:24.938861 29104 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260619 11:07:24.938995 29104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "18e4dd09dbab44b39bfdfda9ca3717ee"
format_stamp: "Formatted at 2026-06-19 11:07:24 on dist-test-slave-8vcz"
server_key: "b67cd284c90d75d1be44d5aa315e8779"
server_key_iv: "c1ae153d41106e44357352f4827624e6"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:24.939146 29104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:24.955323 29104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:24.958906 29104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:24.959168 29104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:24.959900 29104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:24.960954 29104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:24.961076 29104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:24.961194 29104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:24.961266 29104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:24.975525 29104 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:43059
I20260619 11:07:24.976727 29104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:07:24.977092 29235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:43059 every 8 connection(s)
I20260619 11:07:24.978734 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 29104
I20260619 11:07:24.978859 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:07:24.979166 16628 external_mini_cluster.cc:1428] Setting key 9c56f8aee3275ffb946eff801b74ad53
Jun 19 11:07:24 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867244, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
I20260619 11:07:24.994125 29101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:24.980417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:59221 (local address 127.16.61.62:46033)
0619 11:07:24.980597 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:24.980602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:24.981024 (+ 422us) server_negotiation.cc:407] Connection header received
0619 11:07:24.982256 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:24.982261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:24.982312 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:24.982370 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:24.984025 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:24.984849 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:24.986505 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:24.986883 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:24.986992 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:24.990003 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:24.990006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:24.990009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:24.990038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:24.991803 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:24.992397 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:24.992400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:24.992402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:24.992483 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:24.992872 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:24.992876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:24.992877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:24.993078 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:24.993160 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:24.993821 (+ 661us) server_negotiation.cc:299] Negotiation successful
0619 11:07:24.993945 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 11:07:24.995039 29238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:24.979314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46033 (local address 127.16.61.2:59221)
0619 11:07:24.980861 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:24.980918 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:24.982014 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:24.982772 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:24.982782 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:24.983148 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:24.983853 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:24.983876 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:24.984973 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:24.984977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:24.986363 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:24.986374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:24.987158 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:24.987161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:24.987357 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:24.988114 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:24.988134 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:24.989814 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:24.991959 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:24.991965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:24.991979 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:24.992279 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:24.992586 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:24.992589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:24.992591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:24.992752 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:24.993228 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:24.993234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:24.993532 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 11:07:24.993727 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 11:07:24.994050 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:07:24.996246 29236 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46033
I20260619 11:07:24.996469 29236 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:24.996981 29236 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
I20260619 11:07:24.997959 28884 ts_manager.cc:194] Registered new tserver with Master: 18e4dd09dbab44b39bfdfda9ca3717ee (127.16.61.2:43059)
I20260619 11:07:24.998677 28884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:59221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:07:25.013048 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:50017/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46033
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:25.168655 29241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:25.169088 29241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:25.169209 29241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:25.174854 29241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:25.175048 29241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:25.175179 29241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:07:25.182904 29241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.61.20:50017/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:25.184418 29241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:25.185678 29241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:25.197825 29249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:25.197865 29247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:25.198271 29246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:25.201241 29241 server_base.cc:1034] running on GCE node
I20260619 11:07:25.201701 29241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:25.202401 29241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:25.217936 29241 hybrid_clock.cc:648] HybridClock initialized: now 1781867245217850 us; error 90 us; skew 500 ppm
Jun 19 11:07:25 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:25.221764 29241 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:07:25.222939 29241 webserver.cc:460] Webserver started at http://127.16.61.3:35509/ using document root <none> and password file <none>
I20260619 11:07:25.223618 29241 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:25.223716 29241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:25.223932 29241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 11:07:25 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:25.241715 29241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "f999b415b63748108a209a37aac7ce3e"
format_stamp: "Formatted at 2026-06-19 11:07:25 on dist-test-slave-8vcz"
server_key: "ce546dce72299efde7e570771976dcf6"
server_key_iv: "53a014b17bce646cca3cc6cf1f21f418"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:25.242457 29241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "f999b415b63748108a209a37aac7ce3e"
format_stamp: "Formatted at 2026-06-19 11:07:25 on dist-test-slave-8vcz"
server_key: "ce546dce72299efde7e570771976dcf6"
server_key_iv: "53a014b17bce646cca3cc6cf1f21f418"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:25.249194 29241 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260619 11:07:25.269564 29256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:25.271193 29241 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:07:25.271325 29241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "f999b415b63748108a209a37aac7ce3e"
format_stamp: "Formatted at 2026-06-19 11:07:25 on dist-test-slave-8vcz"
server_key: "ce546dce72299efde7e570771976dcf6"
server_key_iv: "53a014b17bce646cca3cc6cf1f21f418"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:25.271462 29241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:25.289167 29241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:25.292737 29241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:25.292930 29241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:25.293507 29241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:25.294567 29241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:25.294626 29241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:25.294679 29241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:25.294708 29241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:25.306876 29241 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:36519
I20260619 11:07:25.307909 29241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:07:25.308383 29369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:36519 every 8 connection(s)
I20260619 11:07:25.312541 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 29241
I20260619 11:07:25.312654 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:07:25.312944 16628 external_mini_cluster.cc:1428] Setting key e47e47e45803b4d7cdcf5a5d335cf6dc
Jun 19 11:07:25 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:25.329245 29101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:25.312123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:52055 (local address 127.16.61.62:46033)
0619 11:07:25.312250 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:25.312256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:25.312824 (+ 568us) server_negotiation.cc:407] Connection header received
0619 11:07:25.315634 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:25.315638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:25.315722 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:25.315776 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:25.317611 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:25.318468 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:25.320230 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:25.320547 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:25.320613 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:25.323934 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:25.323937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:25.323940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:25.323992 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:25.325647 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:25.327466 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:25.327470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:25.327472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:25.327553 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:25.328042 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:25.328045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:25.328047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:25.328245 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:25.328320 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:25.328964 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 11:07:25.329089 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:07:25.330739 29372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:25.311075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46033 (local address 127.16.61.3:52055)
0619 11:07:25.312662 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:25.312713 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:25.315395 (+ 2682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:25.316200 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:25.316209 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:25.316555 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:25.317203 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:25.317221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:25.318624 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:25.318627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:25.320087 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:25.320101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:25.320818 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:25.320823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:25.321003 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:25.321629 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:25.321648 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:25.323621 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:25.326273 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:25.326279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:25.326294 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:25.327358 (+ 1064us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:25.327647 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:25.327650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:25.327652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:25.327943 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:25.328383 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:25.328390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:25.328696 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 11:07:25.329430 (+ 734us) client_negotiation.cc:240] Negotiation successful
0619 11:07:25.329743 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"thread_start_us":133,"threads_started":1}
I20260619 11:07:25.331871 29370 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46033
I20260619 11:07:25.332110 29370 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:25.332619 29370 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
I20260619 11:07:25.333572 28884 ts_manager.cc:194] Registered new tserver with Master: f999b415b63748108a209a37aac7ce3e (127.16.61.3:36519)
I20260619 11:07:25.334252 28884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:52055
I20260619 11:07:25.338626 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:07:25.339356 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 28967
I20260619 11:07:25.348724 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 29104
I20260619 11:07:25.357074 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 29241
I20260619 11:07:25.364130 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 28854
I20260619 11:07:25.372393 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:46033
--webserver_interface=127.16.61.62
--webserver_port=41823
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:46033
--ranger_config_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:50017/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.61.20:50017/kms with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:25.477555 29373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:25.477778 29373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:25.477851 29373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:25.481844 29373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:25.481897 29373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:25.481917 29373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:25.481933 29373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:25.481988 29373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:25.487037 29373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.61.20:50017/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:46033
--ranger_config_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:46033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=41823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:25.487928 29373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:25.488835 29373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:25.496703 29378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:25.496662 29381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:25.496863 29373 server_base.cc:1034] running on GCE node
W20260619 11:07:25.496671 29379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:25.497560 29373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:25.498513 29373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:25.499730 29373 hybrid_clock.cc:648] HybridClock initialized: now 1781867245499704 us; error 43 us; skew 500 ppm
Jun 19 11:07:25 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:25.502343 29373 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:07:25.503156 29373 webserver.cc:460] Webserver started at http://127.16.61.62:41823/ using document root <none> and password file <none>
I20260619 11:07:25.503644 29373 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:25.503777 29373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:07:25 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:25.584712 29373 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:07:25.587013 29389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:25.588490 29373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 11:07:25.588631 29373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "b9ea57336f824299bb98dc5a79a04128"
format_stamp: "Formatted at 2026-06-19 11:07:23 on dist-test-slave-8vcz"
server_key: "49d25c28d0f0c859c0feb663c6ec1634"
server_key_iv: "eba6d9ea6eab1941b696982eab9350c7"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:25.589192 29373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:25.607465 29373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:25.609927 29373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:25.610114 29373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:25.617122 29373 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:46033
I20260619 11:07:25.617172 29441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:46033 every 8 connection(s)
I20260619 11:07:25.618165 29373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:07:25.618964 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 29373
I20260619 11:07:25.619392 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:50017/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:46007
--local_ip_for_outbound_sockets=127.16.61.1
--tserver_master_addrs=127.16.61.62:46033
--webserver_port=45743
--webserver_interface=127.16.61.1
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.61.20:50017/kms with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:07:25.622370 29442 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:07:25.623957 29442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Bootstrap starting.
I20260619 11:07:25.639850 29442 log.cc:824] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:25.645906 29442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:07:25.646281 29442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Bootstrap complete.
I20260619 11:07:25.650250 29442 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } }
I20260619 11:07:25.650708 29442 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ea57336f824299bb98dc5a79a04128, State: Initialized, Role: FOLLOWER
I20260619 11:07:25.651160 29442 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } }
I20260619 11:07:25.651297 29442 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:07:25.651389 29442 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:07:25.651501 29442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:07:25.653244 29442 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } }
I20260619 11:07:25.653568 29442 leader_election.cc:302] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ea57336f824299bb98dc5a79a04128; no voters:
I20260619 11:07:25.653863 29442 leader_election.cc:288] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:07:25.653990 29446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:07:25.654227 29446 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [term 2 LEADER]: Becoming Leader. State: Replica: b9ea57336f824299bb98dc5a79a04128, State: Running, Role: LEADER
I20260619 11:07:25.654512 29446 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } }
I20260619 11:07:25.654949 29442 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:07:25.655411 29446 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ea57336f824299bb98dc5a79a04128. Latest consensus state: current_term: 2 leader_uuid: "b9ea57336f824299bb98dc5a79a04128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } } }
I20260619 11:07:25.655519 29446 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:25.655874 29450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:07:25.656008 29447 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b9ea57336f824299bb98dc5a79a04128" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ea57336f824299bb98dc5a79a04128" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 46033 } } }
I20260619 11:07:25.656167 29447 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:25.658742 29450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:07:25.659238 29442 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 11:07:25.659320 29450 catalog_manager.cc:1259] Loaded cluster ID: be7a93855f354eaaa734e5b1dbb00207
I20260619 11:07:25.659392 29450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:07:25.661669 29450 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:07:25.662498 29450 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Loaded TSK: 0
I20260619 11:07:25.663807 29450 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:07:25.734009 29444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:25.734318 29444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:25.734402 29444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:25.739200 29444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:25.739276 29444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:25.739378 29444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:07:25.744854 29444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.61.20:50017/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:46007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=45743
--tserver_master_addrs=127.16.61.62:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:25.745870 29444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:25.747017 29444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:25.756654 29482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:25.756775 29480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:25.756632 29479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:25.757321 29444 server_base.cc:1034] running on GCE node
I20260619 11:07:25.757687 29444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:25.758306 29444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:25.759492 29444 hybrid_clock.cc:648] HybridClock initialized: now 1781867245759459 us; error 41 us; skew 500 ppm
Jun 19 11:07:25 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:25.762912 29444 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:07:25.763954 29444 webserver.cc:460] Webserver started at http://127.16.61.1:45743/ using document root <none> and password file <none>
I20260619 11:07:25.764654 29444 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:25.764802 29444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:07:25 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:25.789299 29444 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:07:25.792452 29490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:25.796829 29444 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260619 11:07:25.796960 29444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "a2c10d5dc19543f9b1c5a2cd16d79251"
format_stamp: "Formatted at 2026-06-19 11:07:24 on dist-test-slave-8vcz"
server_key: "833279b4712712a138abfd69e7dd2407"
server_key_iv: "ff4a52691b77d18e077b84ff127d6cbf"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:25.797508 29444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:25.815469 29444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:25.819844 29444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:25.820108 29444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:25.820804 29444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:25.821830 29444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:25.821892 29444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:25.821951 29444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:25.821978 29444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:25.832360 29444 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:46007
I20260619 11:07:25.833374 29444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:07:25.834194 29603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:46007 every 8 connection(s)
I20260619 11:07:25.839216 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 29444
I20260619 11:07:25.839779 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:50017/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:43059
--local_ip_for_outbound_sockets=127.16.61.2
--tserver_master_addrs=127.16.61.62:46033
--webserver_port=35277
--webserver_interface=127.16.61.2
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.61.20:50017/kms with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:07:25 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:25.854153 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:25.835779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46033 (local address 127.16.61.1:48095)
0619 11:07:25.836334 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:25.836383 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:25.837217 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:25.838561 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:25.838571 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:25.838898 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:25.839635 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:25.839658 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:25.841988 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:25.841993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:25.843453 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:25.843466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:25.844859 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:25.844865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:25.845106 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:25.845834 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:25.845861 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:25.847458 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:25.850323 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:25.850332 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:25.850350 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:25.850714 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:25.851227 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:25.851232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:25.851235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:25.851427 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:25.852357 (+ 930us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:25.852365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:25.852693 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 11:07:25.852904 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 11:07:25.853213 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":143,"threads_started":1}
I20260619 11:07:25.856334 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:25.835626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:48095 (local address 127.16.61.62:46033)
0619 11:07:25.837592 (+ 1966us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:25.837607 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:25.837638 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:07:25.837852 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:25.837860 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:25.838168 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:25.838504 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:25.840111 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:25.841813 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:25.843593 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:25.844434 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:25.844617 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:25.847745 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:25.847754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:25.847774 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:25.847811 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:25.850161 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:25.850864 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:25.850872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:25.850880 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:25.851107 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:25.851550 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:25.851555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:25.851559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:25.852152 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:25.852295 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:25.855162 (+ 2867us) server_negotiation.cc:299] Negotiation successful
0619 11:07:25.855428 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1836,"thread_start_us":123,"threads_started":1}
I20260619 11:07:25.858987 29604 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46033
I20260619 11:07:25.859200 29604 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:25.859721 29604 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
W20260619 11:07:25.861899 29604 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:46033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:07:25.862015 29604 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:25.862161 29604 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
I20260619 11:07:25.863018 29604 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:25.863220 29604 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
W20260619 11:07:25.863961 29604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:07:25.995973 29608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:25.996321 29608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:25.996399 29608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:26.002446 29608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:26.002524 29608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:26.002615 29608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:07:26.008999 29608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.61.20:50017/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:43059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=35277
--tserver_master_addrs=127.16.61.62:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:26.010057 29608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:26.011160 29608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:26.022746 29614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:26.023289 29608 server_base.cc:1034] running on GCE node
W20260619 11:07:26.023499 29616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:26.023594 29613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:26.024085 29608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:26.024852 29608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:26.026052 29608 hybrid_clock.cc:648] HybridClock initialized: now 1781867246026026 us; error 39 us; skew 500 ppm
Jun 19 11:07:26 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:26.029624 29608 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:07:26.030764 29608 webserver.cc:460] Webserver started at http://127.16.61.2:35277/ using document root <none> and password file <none>
I20260619 11:07:26.031473 29608 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:26.031585 29608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:07:26 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:26.052150 29608 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 11:07:26.055233 29624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:26.056874 29608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:07:26.056988 29608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "18e4dd09dbab44b39bfdfda9ca3717ee"
format_stamp: "Formatted at 2026-06-19 11:07:24 on dist-test-slave-8vcz"
server_key: "b67cd284c90d75d1be44d5aa315e8779"
server_key_iv: "c1ae153d41106e44357352f4827624e6"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:26.057668 29608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:26.076969 29608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:26.079807 29608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:26.079998 29608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:26.080598 29608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:26.081471 29608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:26.081540 29608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:26.081610 29608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:26.081655 29608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:26.092451 29608 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:43059
I20260619 11:07:26.093478 29608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:07:26.095319 29737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:43059 every 8 connection(s)
I20260619 11:07:26.097688 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 29608
I20260619 11:07:26.098202 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.61.20:50017/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:36519
--local_ip_for_outbound_sockets=127.16.61.3
--tserver_master_addrs=127.16.61.62:46033
--webserver_port=35509
--webserver_interface=127.16.61.3
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.61.20:50017/kms with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:07:26 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:26.111166 29740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:26.096132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46033 (local address 127.16.61.2:46651)
0619 11:07:26.096577 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:26.096626 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:26.097548 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:26.098319 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:26.098328 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:26.098688 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:26.099356 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:26.099378 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.100313 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.100316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:26.101327 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:26.101336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.102231 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.102237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:26.102479 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:26.103150 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:26.103177 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:26.105092 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:26.107511 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:26.107520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:26.107539 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:26.107958 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:26.108306 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:26.108310 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:26.108314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:26.108521 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:26.109079 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:26.109089 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:26.109459 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 11:07:26.109682 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 11:07:26.110031 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
I20260619 11:07:26.111893 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:26.095753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:46651 (local address 127.16.61.62:46033)
0619 11:07:26.095885 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:26.095890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:26.096718 (+ 828us) server_negotiation.cc:407] Connection header received
0619 11:07:26.097802 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:26.097806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:26.097864 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:26.097916 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:26.099548 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.100182 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.101461 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.101920 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.102033 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:26.105306 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:26.105312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:26.105317 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:26.105354 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:26.107352 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:26.108090 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:26.108095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:26.108097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:26.108190 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:26.108647 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:26.108653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:26.108656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:26.108914 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:26.109019 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:26.111592 (+ 2573us) server_negotiation.cc:299] Negotiation successful
0619 11:07:26.111743 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:07:26.113687 29738 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46033
I20260619 11:07:26.113929 29738 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:26.114522 29738 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
W20260619 11:07:26.118537 29738 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:46033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:07:26.118664 29738 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:26.118832 29738 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
I20260619 11:07:26.119649 29738 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:26.119831 29738 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
W20260619 11:07:26.120568 29738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:07:26.246819 29741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:26.247171 29741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:26.247252 29741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:26.253407 29741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:26.253489 29741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:26.253593 29741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:07:26.259334 29741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.61.20:50017/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:36519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=35509
--tserver_master_addrs=127.16.61.62:46033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:26.260507 29741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:26.261536 29741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:26.272017 29748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:26.272011 29747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:26.272429 29741 server_base.cc:1034] running on GCE node
W20260619 11:07:26.275867 29750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:26.276374 29741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:26.277185 29741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:26.278362 29741 hybrid_clock.cc:648] HybridClock initialized: now 1781867246278329 us; error 50 us; skew 500 ppm
Jun 19 11:07:26 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:26.281289 29741 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:07:26.282240 29741 webserver.cc:460] Webserver started at http://127.16.61.3:35509/ using document root <none> and password file <none>
I20260619 11:07:26.282786 29741 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:26.282889 29741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 11:07:26 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:26.301199 29741 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:07:26.304232 29758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:26.305575 29741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:07:26.305707 29741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "f999b415b63748108a209a37aac7ce3e"
format_stamp: "Formatted at 2026-06-19 11:07:25 on dist-test-slave-8vcz"
server_key: "ce546dce72299efde7e570771976dcf6"
server_key_iv: "53a014b17bce646cca3cc6cf1f21f418"
server_key_version: "kuduclusterkey@0"
I20260619 11:07:26.306393 29741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:26.321611 29741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:26.324901 29741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:26.325093 29741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:26.325762 29741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:26.329219 29741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:26.329326 29741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:26.329428 29741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:26.329502 29741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:26.340171 29741 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:36519
I20260619 11:07:26.340220 29871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:36519 every 8 connection(s)
I20260619 11:07:26.341379 29741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:07:26.346616 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 29741
Jun 19 11:07:26 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:26.358969 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:26.343697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:55273 (local address 127.16.61.62:46033)
0619 11:07:26.343851 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:26.343856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:26.344903 (+ 1047us) server_negotiation.cc:407] Connection header received
0619 11:07:26.346156 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:26.346158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:26.346204 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:26.346267 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:26.348029 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.348847 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.350639 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.350970 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.351074 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:26.354974 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:26.354979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:26.354982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:26.355039 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:26.356815 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:26.357284 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:26.357288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:26.357290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:26.357376 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:26.357779 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:26.357785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:26.357789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:26.357988 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:26.358073 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:26.358670 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 11:07:26.358781 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64}
I20260619 11:07:26.359750 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:26.347283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.16.61.62:46033)
0619 11:07:26.347679 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:26.347688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:26.347701 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:07:26.347817 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:26.347820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:26.347870 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:26.347961 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:26.348658 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.349364 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.351069 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.351479 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.351596 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:26.354249 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:26.354252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:26.354255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:26.354281 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:26.356335 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:26.357662 (+ 1327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:26.357666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:26.357667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:26.357748 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:26.358042 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:26.358046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:26.358048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:26.358281 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:26.358951 (+ 670us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:26.359430 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 11:07:26.359555 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":288,"thread_start_us":102,"threads_started":1}
I20260619 11:07:26.359915 29874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:26.344090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:46033 (local address 127.16.61.3:55273)
0619 11:07:26.344719 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:26.344777 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:26.345903 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:26.346466 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:26.346475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:26.346827 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:26.347843 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:26.347866 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.349019 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.349023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:26.350475 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:26.350486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:26.351235 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:26.351238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:26.351378 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:26.352321 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:26.352348 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:26.354512 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:26.356923 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:26.356927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:26.356938 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:26.357173 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:26.357485 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:26.357488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:26.357490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:26.357675 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:26.358128 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:26.358132 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:26.358430 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 11:07:26.358614 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 11:07:26.358902 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":158,"threads_started":1}
I20260619 11:07:26.361202 29872 heartbeater.cc:347] Connected to a master server at 127.16.61.62:46033
I20260619 11:07:26.361449 29872 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:26.362003 29872 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
W20260619 11:07:26.363430 29872 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:46033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 11:07:26.363564 29872 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:26.363754 29872 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
I20260619 11:07:26.364518 29872 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:26.364687 29872 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
W20260619 11:07:26.365375 29872 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 11:07:26 dist-test-slave-8vcz krb5kdc[28410](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:26.864192 29604 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:26.864454 29604 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
I20260619 11:07:26.865423 29406 ts_manager.cc:194] Registered new tserver with Master: a2c10d5dc19543f9b1c5a2cd16d79251 (127.16.61.1:46007)
I20260619 11:07:26.866743 29406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:48095
Jun 19 11:07:27 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:27.120779 29738 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:27.121052 29738 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
I20260619 11:07:27.121731 29406 ts_manager.cc:194] Registered new tserver with Master: 18e4dd09dbab44b39bfdfda9ca3717ee (127.16.61.2:43059)
I20260619 11:07:27.122462 29406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:46651
I20260619 11:07:27.365598 29872 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:27.365902 29872 heartbeater.cc:511] Master 127.16.61.62:46033 requested a full tablet report, sending...
I20260619 11:07:27.366582 29406 ts_manager.cc:194] Registered new tserver with Master: f999b415b63748108a209a37aac7ce3e (127.16.61.3:36519)
I20260619 11:07:27.367336 29406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:55273
I20260619 11:07:27.370726 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:07:27.388696 29895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.373858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.16.61.62:46033)
0619 11:07:27.374533 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.374538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.374551 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:07:27.374588 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.374591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.374635 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.374720 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:27.375625 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.378271 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.379599 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.379954 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.380055 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.383805 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:27.383810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:27.383824 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:27.383852 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:27.386364 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:27.386872 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:27.386876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:27.386878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:27.386970 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:27.387270 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:27.387274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:27.387276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:27.387496 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:27.387622 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.388378 (+ 756us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.388514 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":74,"threads_started":1}
Jun 19 11:07:27 dist-test-slave-8vcz krb5kdc[28410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for HTTP/127.16.61.20@KRBTEST.COM
I20260619 11:07:27.402335 29406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:07:27.407720 29406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:07:27.437801 29900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.424348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:43059 (local address 127.0.0.1:54880)
0619 11:07:27.427374 (+ 3026us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.427428 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.427555 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.429612 (+ 2057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.429619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.429644 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.429776 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.429783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.434446 (+ 4663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.434449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.436260 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.436270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.437295 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.437299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.437407 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.437452 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.437542 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.437657 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2843,"thread_start_us":92,"threads_started":1}
I20260619 11:07:27.439417 29901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.424659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.16.61.2:43059)
0619 11:07:27.427879 (+ 3220us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.427887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.427911 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:07:27.427988 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.427993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.429262 (+ 1269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.429421 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.429932 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.434316 (+ 4384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.436411 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.437160 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.439031 (+ 1871us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.439159 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.439206 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.439288 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3105,"thread_start_us":96,"threads_started":1}
I20260619 11:07:27.442603 29671 tablet_service.cc:1467] Processing CreateTablet for tablet 09b02378457f4e04a0b6d7d3f7f149a5 (DEFAULT_TABLE table=test-table [id=8b139676f99141d4b9e39e3dd569bfa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:27.443712 29671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b02378457f4e04a0b6d7d3f7f149a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:27.444142 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.425411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:36519 (local address 127.0.0.1:54108)
0619 11:07:27.430670 (+ 5259us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.430682 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.430758 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.431423 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.431426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.431442 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.431536 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.431541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.432905 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.432908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.434308 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.434316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.443743 (+ 9427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.443748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.443844 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.443868 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.443944 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.444011 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5205,"thread_start_us":89,"threads_started":1}
I20260619 11:07:27.445624 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.425560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.16.61.3:36519)
0619 11:07:27.430230 (+ 4670us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.430236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.430853 (+ 617us) server_negotiation.cc:407] Connection header received
0619 11:07:27.430935 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.430940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.431130 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.431259 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.431684 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.432787 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.435726 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.436558 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.436655 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.436794 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.445373 (+ 8579us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.445481 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4560,"thread_start_us":116,"threads_started":1}
I20260619 11:07:27.448256 29788 tablet_service.cc:1467] Processing CreateTablet for tablet 09b02378457f4e04a0b6d7d3f7f149a5 (DEFAULT_TABLE table=test-table [id=8b139676f99141d4b9e39e3dd569bfa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:27.449209 29788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b02378457f4e04a0b6d7d3f7f149a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:27.450201 29902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.425096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:46007 (local address 127.0.0.1:47022)
0619 11:07:27.430398 (+ 5302us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.430414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.430500 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.445083 (+ 14583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.445086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.445104 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.445212 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.445219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.447713 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.447716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.448805 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.448815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.449843 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.449847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.449940 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.449964 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.450013 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.450072 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5239,"thread_start_us":77,"threads_started":1}
I20260619 11:07:27.450888 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.425270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.16.61.1:46007)
0619 11:07:27.444490 (+ 19220us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.444497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.444522 (+ 25us) server_negotiation.cc:407] Connection header received
0619 11:07:27.444596 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.444600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.444787 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.444929 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.446491 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.447558 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.448959 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.449707 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.450500 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.450632 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.450679 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.450760 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19108,"thread_start_us":72,"threads_started":1}
I20260619 11:07:27.453364 29538 tablet_service.cc:1467] Processing CreateTablet for tablet 09b02378457f4e04a0b6d7d3f7f149a5 (DEFAULT_TABLE table=test-table [id=8b139676f99141d4b9e39e3dd569bfa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:27.454270 29538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09b02378457f4e04a0b6d7d3f7f149a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:27.464783 29908 tablet_bootstrap.cc:492] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Bootstrap starting.
I20260619 11:07:27.465299 29909 tablet_bootstrap.cc:492] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: Bootstrap starting.
I20260619 11:07:27.467571 29907 tablet_bootstrap.cc:492] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251: Bootstrap starting.
I20260619 11:07:27.467867 29908 tablet_bootstrap.cc:654] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:27.468776 29908 log.cc:824] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:27.469203 29909 tablet_bootstrap.cc:654] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:27.470731 29909 log.cc:824] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:27.471359 29908 tablet_bootstrap.cc:492] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: No bootstrap required, opened a new log
I20260619 11:07:27.471562 29908 ts_tablet_manager.cc:1397] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 11:07:27.472791 29907 tablet_bootstrap.cc:654] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:27.473553 29907 log.cc:824] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:27.475766 29908 raft_consensus.cc:348] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.476133 29908 raft_consensus.cc:374] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:27.476234 29908 raft_consensus.cc:729] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e4dd09dbab44b39bfdfda9ca3717ee, State: Initialized, Role: FOLLOWER
I20260619 11:07:27.476771 29907 tablet_bootstrap.cc:492] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251: No bootstrap required, opened a new log
I20260619 11:07:27.476794 29908 consensus_queue.cc:260] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.476967 29907 ts_tablet_manager.cc:1397] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 11:07:27.477813 29908 ts_tablet_manager.cc:1428] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:07:27.478139 29738 heartbeater.cc:503] Master 127.16.61.62:46033 was elected leader, sending a full tablet report...
I20260619 11:07:27.480649 29907 raft_consensus.cc:348] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.480891 29907 raft_consensus.cc:374] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:27.480934 29907 raft_consensus.cc:729] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c10d5dc19543f9b1c5a2cd16d79251, State: Initialized, Role: FOLLOWER
I20260619 11:07:27.481330 29907 consensus_queue.cc:260] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.481935 29604 heartbeater.cc:503] Master 127.16.61.62:46033 was elected leader, sending a full tablet report...
I20260619 11:07:27.482218 29907 ts_tablet_manager.cc:1428] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:07:27.484736 29909 tablet_bootstrap.cc:492] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: No bootstrap required, opened a new log
I20260619 11:07:27.484936 29909 ts_tablet_manager.cc:1397] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20260619 11:07:27.488668 29909 raft_consensus.cc:348] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.488977 29909 raft_consensus.cc:374] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:27.489068 29909 raft_consensus.cc:729] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f999b415b63748108a209a37aac7ce3e, State: Initialized, Role: FOLLOWER
I20260619 11:07:27.489547 29909 consensus_queue.cc:260] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.490513 29909 ts_tablet_manager.cc:1428] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:07:27.490859 29872 heartbeater.cc:503] Master 127.16.61.62:46033 was elected leader, sending a full tablet report...
W20260619 11:07:27.592918 29605 tablet.cc:2369] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:07:27.593791 29873 tablet.cc:2369] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:07:27.595372 29739 tablet.cc:2369] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:07:27.862922 29913 raft_consensus.cc:482] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:27.863181 29913 raft_consensus.cc:504] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.863514 29915 raft_consensus.cc:482] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:27.863723 29915 raft_consensus.cc:504] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.864571 29913 leader_election.cc:288] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c10d5dc19543f9b1c5a2cd16d79251 (127.16.61.1:46007), f999b415b63748108a209a37aac7ce3e (127.16.61.3:36519)
I20260619 11:07:27.864661 29915 leader_election.cc:288] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18e4dd09dbab44b39bfdfda9ca3717ee (127.16.61.2:43059), a2c10d5dc19543f9b1c5a2cd16d79251 (127.16.61.1:46007)
I20260619 11:07:27.869314 29914 raft_consensus.cc:482] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:27.869472 29914 raft_consensus.cc:504] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.870363 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.864966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:43059 (local address 127.16.61.3:48377)
0619 11:07:27.865659 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.865673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.865755 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.866454 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.866457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.866475 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.866626 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.866632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.867705 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.867708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.869018 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.869029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.869893 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.869896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.869986 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.870020 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.870090 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.870154 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":121,"threads_started":1}
I20260619 11:07:27.870536 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.865840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:36831 (local address 127.16.61.3:36519)
0619 11:07:27.865940 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.865944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.866194 (+ 250us) server_negotiation.cc:407] Connection header received
0619 11:07:27.866370 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.866374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.866416 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.866461 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.866900 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.867717 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.869267 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.869829 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.869901 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.869957 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.870340 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.870420 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":34432}
I20260619 11:07:27.870617 29914 leader_election.cc:288] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18e4dd09dbab44b39bfdfda9ca3717ee (127.16.61.2:43059), f999b415b63748108a209a37aac7ce3e (127.16.61.3:36519)
I20260619 11:07:27.870702 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.865611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:54829 (local address 127.16.61.1:46007)
0619 11:07:27.865951 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.865960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.866221 (+ 261us) server_negotiation.cc:407] Connection header received
0619 11:07:27.866557 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.866560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.866609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.866666 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.867064 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.867817 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.869382 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.869985 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.870058 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.870112 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.870457 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.870537 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":103,"threads_started":1}
I20260619 11:07:27.870750 29901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.866034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:48377 (local address 127.16.61.2:43059)
0619 11:07:27.866124 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.866128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.866140 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:07:27.866180 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.866183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.866222 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.866294 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.866747 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.867556 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.869151 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.869679 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.869777 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.869835 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.870513 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.870601 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":36,"spinlock_wait_cycles":15616}
I20260619 11:07:27.871002 29925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.865232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:36519 (local address 127.16.61.2:36831)
0619 11:07:27.866116 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.866128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.866262 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.866646 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.866650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.866668 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.866786 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.866793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.867851 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.867855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.869127 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.869138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.870065 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.870068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.870191 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.870211 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.870799 (+ 588us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.870872 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":119,"threads_started":1}
I20260619 11:07:27.871589 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.865351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:46007 (local address 127.16.61.3:54829)
0619 11:07:27.866149 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.866163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.866425 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.866798 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.866801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.866817 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.866954 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.866960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.867938 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.867941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.869102 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.869279 (+ 177us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.870197 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.870200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.870332 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.870353 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.871352 (+ 999us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.871437 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":87,"threads_started":1}
I20260619 11:07:27.872021 29823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f999b415b63748108a209a37aac7ce3e" is_pre_election: true
I20260619 11:07:27.872231 29823 raft_consensus.cc:2454] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18e4dd09dbab44b39bfdfda9ca3717ee in term 0.
I20260619 11:07:27.872264 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "f999b415b63748108a209a37aac7ce3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" is_pre_election: true
I20260619 11:07:27.872433 29556 raft_consensus.cc:2454] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f999b415b63748108a209a37aac7ce3e in term 0.
I20260619 11:07:27.871598 29690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "f999b415b63748108a209a37aac7ce3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" is_pre_election: true
I20260619 11:07:27.872838 29762 leader_election.cc:302] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c10d5dc19543f9b1c5a2cd16d79251, f999b415b63748108a209a37aac7ce3e; no voters:
I20260619 11:07:27.872982 29690 raft_consensus.cc:2454] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f999b415b63748108a209a37aac7ce3e in term 0.
I20260619 11:07:27.873111 29915 raft_consensus.cc:2781] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:07:27.873179 29915 raft_consensus.cc:482] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:07:27.873229 29915 raft_consensus.cc:3037] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:27.873522 29628 leader_election.cc:302] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18e4dd09dbab44b39bfdfda9ca3717ee, f999b415b63748108a209a37aac7ce3e; no voters:
I20260619 11:07:27.873751 29913 raft_consensus.cc:2781] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:07:27.873806 29913 raft_consensus.cc:482] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:07:27.873842 29913 raft_consensus.cc:3037] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:27.874459 29915 raft_consensus.cc:504] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.874781 29915 leader_election.cc:288] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [CANDIDATE]: Term 1 election: Requested vote from peers 18e4dd09dbab44b39bfdfda9ca3717ee (127.16.61.2:43059), a2c10d5dc19543f9b1c5a2cd16d79251 (127.16.61.1:46007)
I20260619 11:07:27.874977 29913 raft_consensus.cc:504] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.875278 29913 leader_election.cc:288] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [CANDIDATE]: Term 1 election: Requested vote from peers a2c10d5dc19543f9b1c5a2cd16d79251 (127.16.61.1:46007), f999b415b63748108a209a37aac7ce3e (127.16.61.3:36519)
I20260619 11:07:27.875469 29901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.871219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:52001 (local address 127.16.61.2:43059)
0619 11:07:27.871313 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.871317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.871438 (+ 121us) server_negotiation.cc:407] Connection header received
0619 11:07:27.871616 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.871620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.871695 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.871743 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.872118 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.872736 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.874162 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.874694 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.874768 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.874826 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.875191 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.875276 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:07:27.875555 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.871125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:43059 (local address 127.16.61.1:52001)
0619 11:07:27.871364 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.871377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.871505 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.871855 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.871857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.871876 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.871988 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.871994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.872862 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.872865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.874008 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.874020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.874936 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.874939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.875054 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.875077 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.875307 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.875396 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260619 11:07:27.875795 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "f999b415b63748108a209a37aac7ce3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251"
I20260619 11:07:27.875912 29556 raft_consensus.cc:3037] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:27.877101 29556 raft_consensus.cc:2454] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f999b415b63748108a209a37aac7ce3e in term 1.
I20260619 11:07:27.877609 29690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" is_pre_election: true
I20260619 11:07:27.877764 29690 raft_consensus.cc:2379] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a2c10d5dc19543f9b1c5a2cd16d79251 in current term 1: Already voted for candidate 18e4dd09dbab44b39bfdfda9ca3717ee in this term.
I20260619 11:07:27.878351 29762 leader_election.cc:302] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c10d5dc19543f9b1c5a2cd16d79251, f999b415b63748108a209a37aac7ce3e; no voters:
I20260619 11:07:27.878566 29915 raft_consensus.cc:2781] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:27.878743 29915 raft_consensus.cc:686] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 1 LEADER]: Becoming Leader. State: Replica: f999b415b63748108a209a37aac7ce3e, State: Running, Role: LEADER
I20260619 11:07:27.879050 29915 consensus_queue.cc:237] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } }
I20260619 11:07:27.879274 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.865007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:36399 (local address 127.16.61.1:46007)
0619 11:07:27.865632 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.865637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.865651 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:07:27.865709 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.865715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.865765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.865836 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.875871 (+ 10035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.876613 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.878080 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.878657 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.878728 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.878773 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.879111 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.879177 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546}
I20260619 11:07:27.879468 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.872623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:44373 (local address 127.16.61.3:36519)
0619 11:07:27.872706 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.872710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.875775 (+ 3065us) server_negotiation.cc:407] Connection header received
0619 11:07:27.875944 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.875947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.875988 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.876050 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:27.876383 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.877045 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.878340 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.878886 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.878960 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.879005 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.879321 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.879380 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:07:27.879611 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.864828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:46007 (local address 127.16.61.2:36399)
0619 11:07:27.865215 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.865233 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.865347 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.866269 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.866272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.866290 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.875746 (+ 9456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.875757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.876748 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.876751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.877934 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.877943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.878861 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.878864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.878957 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.878978 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.879422 (+ 444us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.879486 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":9264,"thread_start_us":116,"threads_started":1}
I20260619 11:07:27.879807 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.871454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:36519 (local address 127.16.61.1:44373)
0619 11:07:27.875708 (+ 4254us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.875722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:27.875831 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:27.876173 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:27.876176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:27.876193 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:27.876269 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.876274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.877181 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.877183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.878211 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:27.878220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.879109 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.879112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:27.879190 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.879211 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:07:27.879612 (+ 401us) client_negotiation.cc:240] Negotiation successful
0619 11:07:27.879698 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4185,"thread_start_us":95,"threads_started":1}
I20260619 11:07:27.880137 29823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f999b415b63748108a209a37aac7ce3e" is_pre_election: true
I20260619 11:07:27.880744 29558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251"
I20260619 11:07:27.880820 29823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f999b415b63748108a209a37aac7ce3e"
I20260619 11:07:27.880888 29558 raft_consensus.cc:2379] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 18e4dd09dbab44b39bfdfda9ca3717ee in current term 1: Already voted for candidate f999b415b63748108a209a37aac7ce3e in this term.
I20260619 11:07:27.881338 29690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "f999b415b63748108a209a37aac7ce3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee"
I20260619 11:07:27.881328 29494 leader_election.cc:302] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2c10d5dc19543f9b1c5a2cd16d79251; no voters: 18e4dd09dbab44b39bfdfda9ca3717ee, f999b415b63748108a209a37aac7ce3e
I20260619 11:07:27.881450 29690 raft_consensus.cc:2379] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f999b415b63748108a209a37aac7ce3e in current term 1: Already voted for candidate 18e4dd09dbab44b39bfdfda9ca3717ee in this term.
I20260619 11:07:27.881620 29914 raft_consensus.cc:2726] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 11:07:27.881862 29628 leader_election.cc:302] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18e4dd09dbab44b39bfdfda9ca3717ee; no voters: a2c10d5dc19543f9b1c5a2cd16d79251, f999b415b63748108a209a37aac7ce3e
I20260619 11:07:27.882086 29913 raft_consensus.cc:2726] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 11:07:27.882287 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09b02378457f4e04a0b6d7d3f7f149a5" candidate_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" is_pre_election: true
I20260619 11:07:27.882109 29405 catalog_manager.cc:5495] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e reported cstate change: term changed from 0 to 1, leader changed from <none> to f999b415b63748108a209a37aac7ce3e (127.16.61.3). New cstate: current_term: 1 leader_uuid: "f999b415b63748108a209a37aac7ce3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f999b415b63748108a209a37aac7ce3e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36519 } health_report { overall_health: HEALTHY } } }
I20260619 11:07:27.882442 29556 raft_consensus.cc:2379] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18e4dd09dbab44b39bfdfda9ca3717ee in current term 1: Already voted for candidate f999b415b63748108a209a37aac7ce3e in this term.
I20260619 11:07:27.901412 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.897496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.16.61.3:36519)
0619 11:07:27.897613 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:27.897616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:27.897838 (+ 222us) server_negotiation.cc:407] Connection header received
0619 11:07:27.898021 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:27.898024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:27.898074 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:27.898173 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:07:27.898406 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.898967 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.899939 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:27.900186 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:27.900294 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:27.900606 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:07:27.901033 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:07:27.901156 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:27.901218 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 11:07:27.901266 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:07:27.907411 29556 raft_consensus.cc:1261] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 1 FOLLOWER]: Refusing update from remote peer f999b415b63748108a209a37aac7ce3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:27.907469 29690 raft_consensus.cc:1261] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 1 FOLLOWER]: Refusing update from remote peer f999b415b63748108a209a37aac7ce3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:27.908042 29915 consensus_queue.cc:1035] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c10d5dc19543f9b1c5a2cd16d79251" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:07:27.908237 29930 consensus_queue.cc:1035] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18e4dd09dbab44b39bfdfda9ca3717ee" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:07:27.916572 29934 mvcc.cc:205] Tried to move back new op lower bound from 7298528247421558784 to 7298528247315435520. Current Snapshot: MvccSnapshot[applied={T|T < 7298528247421558784}]
I20260619 11:07:27.916778 29935 mvcc.cc:205] Tried to move back new op lower bound from 7298528247421558784 to 7298528247315435520. Current Snapshot: MvccSnapshot[applied={T|T < 7298528247421558784}]
I20260619 11:07:27.923420 29936 mvcc.cc:205] Tried to move back new op lower bound from 7298528247421558784 to 7298528247315435520. Current Snapshot: MvccSnapshot[applied={T|T < 7298528247421558784}]
I20260619 11:07:27.924785 29405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:07:27.924989 29405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:07:27.927791 29405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9ea57336f824299bb98dc5a79a04128: Sending DeleteTablet for 3 replicas of tablet 09b02378457f4e04a0b6d7d3f7f149a5
I20260619 11:07:27.928455 29671 tablet_service.cc:1514] Processing DeleteTablet for tablet 09b02378457f4e04a0b6d7d3f7f149a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:27 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:54880
I20260619 11:07:27.928921 29944 tablet_replica.cc:330] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: stopping tablet replica
I20260619 11:07:27.929272 29944 raft_consensus.cc:2229] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:07:27.929342 29538 tablet_service.cc:1514] Processing DeleteTablet for tablet 09b02378457f4e04a0b6d7d3f7f149a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:27 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:47022
I20260619 11:07:27.929600 29944 raft_consensus.cc:2258] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:27.930171 29788 tablet_service.cc:1514] Processing DeleteTablet for tablet 09b02378457f4e04a0b6d7d3f7f149a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:27 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:54108
I20260619 11:07:27.930387 29945 tablet_replica.cc:330] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251: stopping tablet replica
I20260619 11:07:27.930642 29945 raft_consensus.cc:2229] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:07:27.930922 29945 raft_consensus.cc:2258] T 09b02378457f4e04a0b6d7d3f7f149a5 P a2c10d5dc19543f9b1c5a2cd16d79251 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:27.931205 29946 tablet_replica.cc:330] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: stopping tablet replica
I20260619 11:07:27.931277 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 29444
I20260619 11:07:27.935590 29944 ts_tablet_manager.cc:1905] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:07:27.938354 29944 ts_tablet_manager.cc:1918] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:07:27.938515 29944 log.cc:1196] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/09b02378457f4e04a0b6d7d3f7f149a5
I20260619 11:07:27.938886 29944 ts_tablet_manager.cc:1939] T 09b02378457f4e04a0b6d7d3f7f149a5 P 18e4dd09dbab44b39bfdfda9ca3717ee: Deleting consensus metadata
I20260619 11:07:27.940322 29393 catalog_manager.cc:4841] TS 18e4dd09dbab44b39bfdfda9ca3717ee (127.16.61.2:43059): tablet 09b02378457f4e04a0b6d7d3f7f149a5 (table test-table [id=8b139676f99141d4b9e39e3dd569bfa9]) successfully deleted
I20260619 11:07:27.940665 29946 raft_consensus.cc:2229] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:07:27.941022 29946 raft_consensus.cc:2258] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 11:07:27.941726 29393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:46007 (error 108)
I20260619 11:07:27.941789 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 29608
I20260619 11:07:27.947207 29902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:27.946879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:46007 (local address 127.0.0.1:47034)
0619 11:07:27.947014 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:27.947099 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:46007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 11:07:27.947393 29393 catalog_manager.cc:4584] TS a2c10d5dc19543f9b1c5a2cd16d79251 (127.16.61.1:46007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09b02378457f4e04a0b6d7d3f7f149a5: Network error: Client connection negotiation failed: client connection to 127.16.61.1:46007: connect: Connection refused (error 111)
I20260619 11:07:27.947994 29946 ts_tablet_manager.cc:1905] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:07:27.950709 29946 ts_tablet_manager.cc:1918] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:07:27.950784 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 29741
I20260619 11:07:27.950855 29946 log.cc:1196] T 09b02378457f4e04a0b6d7d3f7f149a5 P f999b415b63748108a209a37aac7ce3e: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/09b02378457f4e04a0b6d7d3f7f149a5
I20260619 11:07:27.955474 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 29373
I20260619 11:07:27.961050 16628 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 11:07:28.305943 16628 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 11:07:28.306052 16628 mini_ranger.cc:66] Stopping Ranger...
I20260619 11:07:28.651316 16628 mini_ranger.cc:68] Stopped Ranger
2026-06-19 11:07:28.651 UTC [28440] LOG: received smart shutdown request
2026-06-19 11:07:28.654 UTC [28440] LOG: background worker "logical replication launcher" (PID 28449) exited with exit code 1
2026-06-19 11:07:28.655 UTC [28444] LOG: shutting down
2026-06-19 11:07:28.802 UTC [28440] LOG: database system is shut down
2026-06-19T11:07:28Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46783 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:07:28 dist-test-slave-8vcz krb5kdc[29956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:07:28 dist-test-slave-8vcz krb5kdc[29956](info): set up 2 sockets
Jun 19 11:07:28 dist-test-slave-8vcz krb5kdc[29956](info): commencing operation
krb5kdc: starting...
W20260619 11:07:29.951895 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:07:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:07:30.068887 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41537
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:41537,127.16.61.61:42555,127.16.61.60:38807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:30.174551 29969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:30.174816 29969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:30.174885 29969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:30.178818 29969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:30.178894 29969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:30.178930 29969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:30.178982 29969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:30.179016 29969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:30.184114 29969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:41537,127.16.61.61:42555,127.16.61.60:38807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:30.185148 29969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:30.186050 29969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:30.194051 29977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:30.194208 29969 server_base.cc:1034] running on GCE node
W20260619 11:07:30.194141 29974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:30.194100 29975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:30.194918 29969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:30.195958 29969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:30.197122 29969 hybrid_clock.cc:648] HybridClock initialized: now 1781867250197094 us; error 39 us; skew 500 ppm
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:30.202255 29969 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:07:30.203032 29969 webserver.cc:460] Webserver started at http://127.16.61.62:42027/ using document root <none> and password file <none>
I20260619 11:07:30.203536 29969 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:30.203591 29969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:30.203830 29969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:30.205883 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "1e05e0f882194e59ad8e4707df0aaa0c"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "dceefd2515a5dbd698891a535556b5e9"
server_key_iv: "cd008ea78f5de7f9d08fa3e6f8b56515"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.206373 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "1e05e0f882194e59ad8e4707df0aaa0c"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "dceefd2515a5dbd698891a535556b5e9"
server_key_iv: "cd008ea78f5de7f9d08fa3e6f8b56515"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.209793 29969 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:07:30.212061 29984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:30.213198 29969 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:07:30.213292 29969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "1e05e0f882194e59ad8e4707df0aaa0c"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "dceefd2515a5dbd698891a535556b5e9"
server_key_iv: "cd008ea78f5de7f9d08fa3e6f8b56515"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.213369 29969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:30.223905 29969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:30.226419 29969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:30.226554 29969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:30.233319 29969 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:41537
I20260619 11:07:30.233345 30036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:41537 every 8 connection(s)
I20260619 11:07:30.234169 29969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:07:30.236699 30037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:30.240171 30037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:07:30.244230 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 29969
I20260619 11:07:30.244400 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:07:30.244689 16628 external_mini_cluster.cc:1428] Setting key f6c4d70f3f8ff1fcb2a330797f7c9fc3
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.61@KRBTEST.COM; defaulting to no policy
I20260619 11:07:30.254611 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.242282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.16.61.62:41537)
0619 11:07:30.242741 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.242760 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.242955 (+ 195us) server_negotiation.cc:407] Connection header received
0619 11:07:30.243935 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.243940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.244115 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.244289 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.245132 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.245952 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.246898 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.247249 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.247412 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.250155 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.250160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.250169 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.250190 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.251823 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.252357 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.252361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.252362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.252456 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.252813 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.252815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.252816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.253145 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.253243 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.253702 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.253860 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
I20260619 11:07:30.254614 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.242282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41537 (local address 127.0.0.1:39982)
0619 11:07:30.242831 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.242865 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.243609 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.244288 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.244294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.244698 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.244972 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.244986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.246079 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.246083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.246782 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.246791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.247408 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.247417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.247602 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.248167 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.248191 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.250008 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.251927 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.251936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.251946 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.252254 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.252554 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.252558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.252561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.252712 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.253268 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.253277 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.253444 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.253641 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.253940 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":134,"spinlock_wait_cycles":48000,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.61@KRBTEST.COM" created.
I20260619 11:07:30.257048 30037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:07:30.257692 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.257465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42924)
0619 11:07:30.257571 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.257643 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 11:07:30.257923 29987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
I20260619 11:07:30.258734 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.258556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42930)
0619 11:07:30.258653 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.258692 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 11:07:30.258929 30037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:42555: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab.
Entry for principal kudu/127.16.61.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab.
I20260619 11:07:30.271275 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.61:42555
--webserver_interface=127.16.61.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab
--principal=kudu/127.16.61.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:41537,127.16.61.61:42555,127.16.61.60:38807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:07:30.308171 30037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } attempt: 1
I20260619 11:07:30.309067 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.308771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42946)
0619 11:07:30.308922 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.308982 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 11:07:30.309839 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.309658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42960)
0619 11:07:30.309750 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.309791 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:07:30.310006 30037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:42555: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
I20260619 11:07:30.378166 30037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } attempt: 2
I20260619 11:07:30.378985 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.378713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42962)
0619 11:07:30.378856 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.378914 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 11:07:30.379137 30044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:30.379446 30044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:30.379528 30044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 11:07:30.379730 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.379481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42966)
0619 11:07:30.379582 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.379631 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 11:07:30.379856 30037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:42555: Network error: Client connection negotiation failed: client connection to 127.16.61.61:42555: connect: Connection refused (error 111)
W20260619 11:07:30.383512 30044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:30.383559 30044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:30.383579 30044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:30.383594 30044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:30.383610 30044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:30.389219 30044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:41537,127.16.61.61:42555,127.16.61.60:38807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab
--principal=kudu/127.16.61.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.61:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.61.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:30.390096 30044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:30.391041 30044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:30.398955 30050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:30.398967 30049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:30.398950 30052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:30.399236 30044 server_base.cc:1034] running on GCE node
I20260619 11:07:30.399878 30044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:30.400892 30044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:30.402071 30044 hybrid_clock.cc:648] HybridClock initialized: now 1781867250402041 us; error 41 us; skew 500 ppm
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:30.404757 30044 init.cc:377] Logged in from keytab as kudu/127.16.61.61@KRBTEST.COM (short username kudu)
I20260619 11:07:30.405467 30044 webserver.cc:460] Webserver started at http://127.16.61.61:46297/ using document root <none> and password file <none>
I20260619 11:07:30.405937 30044 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:30.405993 30044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:30.406147 30044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:30.408004 30044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/data/instance:
uuid: "4332eaf5038a4fb0a522b46bc570cac9"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "9d745fbf9b7ead63457221d3336733b1"
server_key_iv: "d2501d6547a924e078c511898b1824b5"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.408474 30044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/wal/instance:
uuid: "4332eaf5038a4fb0a522b46bc570cac9"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "9d745fbf9b7ead63457221d3336733b1"
server_key_iv: "d2501d6547a924e078c511898b1824b5"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.411759 30044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:07:30.413846 30059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:30.414939 30044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:30.415076 30044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/wal
uuid: "4332eaf5038a4fb0a522b46bc570cac9"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "9d745fbf9b7ead63457221d3336733b1"
server_key_iv: "d2501d6547a924e078c511898b1824b5"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.415177 30044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:30.429325 30044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:30.432344 30044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:30.432534 30044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:30.439262 30044 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.61:42555
I20260619 11:07:30.439369 30111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.61:42555 every 8 connection(s)
I20260619 11:07:30.440189 30044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
I20260619 11:07:30.442734 30112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:30.446241 30112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:07:30.447202 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30044
I20260619 11:07:30.447338 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-1/wal/instance
I20260619 11:07:30.447620 16628 external_mini_cluster.cc:1428] Setting key b75e7595b15487496f580bf9194d199b
WARNING: no policy specified for kudu/127.16.61.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.60@KRBTEST.COM" created.
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:30.462110 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.448194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.16.61.62:41537)
0619 11:07:30.448352 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.448358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.449215 (+ 857us) server_negotiation.cc:407] Connection header received
0619 11:07:30.450088 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.450091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.450152 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.450254 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.451404 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.451981 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.453080 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.453317 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.453440 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.458417 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.458421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.458422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.458447 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.459981 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.460508 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.460514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.460519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.460603 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.460970 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.460973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.460975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.461156 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.461264 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.461804 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.461939 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:07:30.462893 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.448543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41537 (local address 127.0.0.1:39986)
0619 11:07:30.449046 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.449110 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.449889 (+ 779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.450422 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.450429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.450736 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.451256 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.451272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.452107 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.452110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.452955 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.452964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.453474 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.453478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.453639 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.454250 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.454266 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.458225 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.460099 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.460104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.460116 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.460368 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.460699 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.460703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.460705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.460860 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.461276 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.461284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.461548 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.461742 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.462051 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260619 11:07:30.464463 30112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
I20260619 11:07:30.471007 30037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab.
Entry for principal kudu/127.16.61.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab.
I20260619 11:07:30.474529 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.464967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42976)
0619 11:07:30.465070 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.465081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.465140 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.465850 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.465852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.465974 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.466070 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.466075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.467234 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.467240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.467954 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.467964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.468660 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.468663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.468735 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.469112 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.469122 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.470372 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.472760 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.472765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.472767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.472949 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.473285 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.473287 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.473289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.473365 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.473910 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.473916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.474046 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.474164 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.474284 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 11:07:30.474594 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.60:38807
--webserver_interface=127.16.61.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab
--principal=kudu/127.16.61.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:41537,127.16.61.61:42555,127.16.61.60:38807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:07:30.474644 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.465128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.16.61.61:42555)
0619 11:07:30.465450 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.465458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.465474 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:07:30.465505 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.465509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.465645 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.465853 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.466198 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.467098 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.468201 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.468554 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.468638 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.470493 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.470501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.470510 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.470530 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.472675 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.473054 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.473079 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.473083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.473200 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.473498 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.473503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.473504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.473798 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.473937 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.474251 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.474385 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":149,"threads_started":1}
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
I20260619 11:07:30.477141 30112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:07:30.477902 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.477583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55308)
0619 11:07:30.477752 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.477825 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260619 11:07:30.478076 30060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
I20260619 11:07:30.479177 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.478895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55318)
0619 11:07:30.479023 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.479090 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 11:07:30.479324 30112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:38807: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
I20260619 11:07:30.480798 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.471461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42978)
0619 11:07:30.471579 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.471596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.471703 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.472050 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.472055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.472218 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.472348 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.472364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.473114 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.473120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.474393 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.474403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.475488 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.475492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.475595 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.476077 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.476096 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.477333 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.479269 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.479274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.479276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.479534 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.479838 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.479842 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.479844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.479935 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.480280 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.480283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.480378 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.480480 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.480614 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 11:07:30.480840 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.471517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42978 (local address 127.16.61.61:42555)
0619 11:07:30.471856 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.471862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.471873 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:07:30.471899 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.471901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.471931 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.472000 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.472480 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.473016 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.474525 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.475368 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.475486 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.477472 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.477476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.477480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.477503 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.479163 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.479634 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.479639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.479641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.479742 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.480022 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.480027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.480029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.480186 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.480285 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.480587 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.480694 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
I20260619 11:07:30.481382 30037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:07:30.482088 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.481787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55334)
0619 11:07:30.481925 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.481990 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 11:07:30.482800 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.482502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55342)
0619 11:07:30.482658 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.482719 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 11:07:30.482933 30037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:38807: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
I20260619 11:07:30.514098 30037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } attempt: 1
I20260619 11:07:30.514825 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.514546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55356)
0619 11:07:30.514690 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.514751 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 11:07:30.515504 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.515293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55360)
0619 11:07:30.515390 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.515437 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:07:30.515693 30037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:38807: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
I20260619 11:07:30.528573 30112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } attempt: 1
I20260619 11:07:30.529248 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.528979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55372)
0619 11:07:30.529117 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.529174 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 11:07:30.529929 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.529732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55374)
0619 11:07:30.529827 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.529874 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 11:07:30.530057 30112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:38807: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
W20260619 11:07:30.583431 30119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:30.583710 30119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:30.583770 30119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:30.587574 30119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:30.587623 30119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:30.587642 30119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:30.587658 30119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:30.587733 30119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:30.590886 30037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } attempt: 2
I20260619 11:07:30.591686 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.591366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55380)
0619 11:07:30.591515 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.591584 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 11:07:30.592376 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.592168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55394)
0619 11:07:30.592270 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.592319 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:07:30.592533 30037 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:38807: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
I20260619 11:07:30.593447 30119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:41537,127.16.61.61:42555,127.16.61.60:38807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab
--principal=kudu/127.16.61.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.60:38807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.61.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:30.594398 30119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:30.595358 30119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:07:30.598217 30112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } attempt: 2
I20260619 11:07:30.598848 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.598589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55404)
0619 11:07:30.598735 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.598786 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 11:07:30.599670 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.599390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55408)
0619 11:07:30.599527 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.599604 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 11:07:30.599831 30112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:38807: Network error: Client connection negotiation failed: client connection to 127.16.61.60:38807: connect: Connection refused (error 111)
W20260619 11:07:30.603406 30126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:30.603464 30125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:30.603444 30128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:30.603765 30119 server_base.cc:1034] running on GCE node
I20260619 11:07:30.604290 30119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:30.605566 30119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:30.606750 30119 hybrid_clock.cc:648] HybridClock initialized: now 1781867250606726 us; error 38 us; skew 500 ppm
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:30.609428 30119 init.cc:377] Logged in from keytab as kudu/127.16.61.60@KRBTEST.COM (short username kudu)
I20260619 11:07:30.610256 30119 webserver.cc:460] Webserver started at http://127.16.61.60:42315/ using document root <none> and password file <none>
I20260619 11:07:30.610842 30119 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:30.610930 30119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:30.611119 30119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:30.613389 30119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/data/instance:
uuid: "30d221b0b76642c6a78b8672eb0de505"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "7160450a8d310e3c8f282136a2619ac6"
server_key_iv: "cd525535feda8dd2b0fe11b27f77e0a6"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.613924 30119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/wal/instance:
uuid: "30d221b0b76642c6a78b8672eb0de505"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "7160450a8d310e3c8f282136a2619ac6"
server_key_iv: "cd525535feda8dd2b0fe11b27f77e0a6"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.617395 30119 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:07:30.619855 30135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:30.621112 30119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:07:30.621250 30119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/wal
uuid: "30d221b0b76642c6a78b8672eb0de505"
format_stamp: "Formatted at 2026-06-19 11:07:30 on dist-test-slave-8vcz"
server_key: "7160450a8d310e3c8f282136a2619ac6"
server_key_iv: "cd525535feda8dd2b0fe11b27f77e0a6"
server_key_version: "encryptionkey@0"
I20260619 11:07:30.621357 30119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:30.647264 30119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:30.650065 30119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:30.650261 30119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:30.657310 30119 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.60:38807
I20260619 11:07:30.657341 30187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.60:38807 every 8 connection(s)
I20260619 11:07:30.658233 30119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
I20260619 11:07:30.660835 30188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:30.661348 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30119
I20260619 11:07:30.661490 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781867094678158-16628-0/minicluster-data/master-2/wal/instance
I20260619 11:07:30.661778 16628 external_mini_cluster.cc:1428] Setting key 5b4a6f20a71b2416a5020b1c884bb0ec
I20260619 11:07:30.664825 30188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:30.675019 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.666466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.16.61.62:41537)
0619 11:07:30.666597 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.666602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.666858 (+ 256us) server_negotiation.cc:407] Connection header received
0619 11:07:30.667034 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.667037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.667087 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.667176 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.667843 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.668313 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.669361 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.669581 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.669692 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.671944 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.671947 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.671949 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.671970 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.673299 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.673769 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.673772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.673773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.673844 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.674114 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.674119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.674121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.674305 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.674411 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.674739 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.674850 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:07:30.675745 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.663060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.16.61.60:38807)
0619 11:07:30.663481 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.663495 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.663528 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:07:30.664202 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.664217 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.664529 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.664843 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.665655 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.666979 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.667935 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.668399 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.668563 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.671148 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.671154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.671163 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.671183 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.673014 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.673436 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.673448 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.673449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.673575 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.673852 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.673857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.673858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.674197 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.674404 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.674632 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.674999 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260619 11:07:30.675748 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.666349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41537 (local address 127.0.0.1:39992)
0619 11:07:30.666742 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.666778 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.666915 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.667223 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.667228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.667552 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.667715 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.667730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.668434 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.668440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.669239 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.669249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.669722 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.669726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.669835 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.670347 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.670370 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.671802 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.673404 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.673414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.673430 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.673657 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.673930 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.673934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.673939 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.674026 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.674408 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.674414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.674550 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.674686 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.674934 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":150,"threads_started":1}
I20260619 11:07:30.676956 30188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:30.686174 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.677402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:42555 (local address 127.0.0.1:42988)
0619 11:07:30.677558 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.677573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.677636 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.677930 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.677934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.678101 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.678190 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.678199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.679161 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.679164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.679875 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.679884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.680473 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.680476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.680573 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.681028 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.681039 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.682511 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.684720 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.684723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.684724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.684908 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.685204 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.685207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.685209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.685304 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.685740 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.685742 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.685814 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.685893 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.686022 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 11:07:30.686264 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.677532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.16.61.61:42555)
0619 11:07:30.677676 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.677681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.677697 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:07:30.677762 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.677766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.677811 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.677898 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.678319 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.679048 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.679982 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.680364 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.680469 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.682669 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.682672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.682675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.682698 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.684608 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.685026 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.685029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.685031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.685113 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.685404 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.685408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.685409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.685628 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.685737 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.685975 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.686100 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:07:30.686602 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.677845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.16.61.62:41537)
0619 11:07:30.677975 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.677979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.677991 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:07:30.678022 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.678025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.678064 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.678144 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.678821 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.679419 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.680404 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.680741 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.680854 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.683265 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.683269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.683271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.683301 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.684901 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.685330 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.685334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.685336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.685445 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.685707 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.685712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.685713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.685921 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.686125 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.686259 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.686436 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 11:07:30.687358 30188 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:07:30.691026 30112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } attempt: 3
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
I20260619 11:07:30.691728 30037 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } attempt: 3
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
I20260619 11:07:30.695472 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.687833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55424)
0619 11:07:30.688019 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.688033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.688102 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.688333 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.688335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.688487 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.688549 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.688553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.689247 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.689253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.689740 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.689747 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.690189 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.690192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.690307 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.690705 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.690716 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.691950 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.693834 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.693839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.693843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.694073 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.694427 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.694430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.694431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.694517 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.695024 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.695026 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.695110 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.695203 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.695314 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":21}
I20260619 11:07:30.695941 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.688061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.16.61.60:38807)
0619 11:07:30.688172 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.688176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.688187 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:07:30.688212 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.688214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.688249 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.688323 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.688654 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.689149 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.689859 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.690091 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.690191 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.692086 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.692089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.692091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.692113 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.693745 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.694167 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.694171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.694174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.694263 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.694607 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.694609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.694611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.694814 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.694983 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.695652 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.695788 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":43}
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
I20260619 11:07:30.698844 30188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505: Bootstrap starting.
I20260619 11:07:30.701105 30188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:30.701469 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.692105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55444)
0619 11:07:30.692250 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.692263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.692597 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.693136 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.693139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.693281 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.693356 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.693361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.694097 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.694102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.695562 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.695575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.696259 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.696265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.696358 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.697017 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.697031 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.698191 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.700034 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.700039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.700041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.700238 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.700525 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.700529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.700531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.700614 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.700994 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.700997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.701093 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.701196 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.701320 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 11:07:30.701530 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.692321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.16.61.60:38807)
0619 11:07:30.692548 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.692559 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.692572 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:07:30.692709 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.692712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.692757 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.692855 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.693472 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.693987 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.695723 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.696121 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.696935 (+ 814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.698319 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.698322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.698325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.698356 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.699927 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.700351 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.700355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.700356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.700435 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.700711 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.700714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.700716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.700899 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.701001 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.701255 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.701363 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260619 11:07:30.702118 30188 log.cc:824] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:07:30 dist-test-slave-8vcz krb5kdc[29956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
I20260619 11:07:30.704253 30188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505: No bootstrap required, opened a new log
I20260619 11:07:30.704473 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: Bootstrap starting.
I20260619 11:07:30.707033 30037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:30.707093 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.691437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:38807 (local address 127.0.0.1:55438)
0619 11:07:30.691623 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:30.691655 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:30.692021 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:30.693083 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:30.693088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:30.693266 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:30.693365 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.693374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.694939 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.694944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.695464 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:30.695472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.696337 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.696341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:30.696419 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.696860 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:30.696874 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:30.703877 (+ 7003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:30.705663 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.705666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.705668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.705868 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.706160 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:30.706163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:30.706165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:30.706248 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:30.706601 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:30.706604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:30.706700 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 11:07:30.706806 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 11:07:30.706934 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 11:07:30.707134 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.691510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.16.61.60:38807)
0619 11:07:30.692858 (+ 1348us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.692867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.692877 (+ 10us) server_negotiation.cc:407] Connection header received
0619 11:07:30.692907 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.692910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.692946 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.693495 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.693640 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.694833 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.695829 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.696200 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.696615 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.704001 (+ 7386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.704005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.704007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.704030 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.705549 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.705967 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.705972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.705973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.706063 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.706336 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.706341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.706345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.706506 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.706607 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.706863 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.706963 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918,"thread_start_us":106,"threads_started":1}
I20260619 11:07:30.708151 30037 log.cc:824] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:30.708992 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.687922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.16.61.61:42555)
0619 11:07:30.688028 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.688032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.688044 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:07:30.688076 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.688080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.688123 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.688195 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.688766 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.689244 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.689966 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.690196 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.690296 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.693899 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.693906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.693910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.693935 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.702783 (+ 8848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.707721 (+ 4938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.707724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.707727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.707818 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.708127 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.708130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.708132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.708351 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.708561 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.708708 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.708822 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:07:30.710099 30037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: No bootstrap required, opened a new log
I20260619 11:07:30.710233 30112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9: Bootstrap starting.
I20260619 11:07:30.712299 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:07:30.712795 30112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:30.713577 30112 log.cc:824] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:30.714141 30037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.714496 30037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:30.714612 30037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e05e0f882194e59ad8e4707df0aaa0c, State: Initialized, Role: FOLLOWER
I20260619 11:07:30.715272 30037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.715855 30188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.716094 30188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:30.716153 30188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d221b0b76642c6a78b8672eb0de505, State: Initialized, Role: FOLLOWER
I20260619 11:07:30.716584 30188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.716861 30037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:07:30.717036 30112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9: No bootstrap required, opened a new log
I20260619 11:07:30.717921 30188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:07:30.722373 30222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:07:30.722762 30222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:07:30.723106 30206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.723315 30206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:07:30.723767 30205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.723933 30205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:07:30.721719 29969 master_runner.cc:186] Error getting master registration for 127.16.61.61:42555: OK, instance_id { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" instance_seqno: 1781867250437470 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:07:30.725551 29969 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:07:30.725957 30112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.726224 30112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:30.726316 30112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4332eaf5038a4fb0a522b46bc570cac9, State: Initialized, Role: FOLLOWER
I20260619 11:07:30.726805 30112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.724207 30119 master_runner.cc:186] Error getting master registration for 127.16.61.61:42555: OK, instance_id { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" instance_seqno: 1781867250437470 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:07:30.727591 30119 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 11:07:30.727855 29990 tablet.cc:2369] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:07:30.728191 30112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 11:07:30.731287 30228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:07:30.731405 30228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:07:30.731581 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.714605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55448 (local address 127.16.61.60:38807)
0619 11:07:30.714710 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.714716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.721178 (+ 6462us) server_negotiation.cc:407] Connection header received
0619 11:07:30.721216 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.721219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.721261 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.721332 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.722127 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.722864 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.723998 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.724357 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.724496 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.726391 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.726397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.726400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.726426 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.729297 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.730039 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.730043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.730044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.730129 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.730440 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.730444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.730445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.730640 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.730785 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.731312 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.731428 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:07:30.732309 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.714836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.16.61.62:41537)
0619 11:07:30.714951 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.714956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.714969 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:07:30.715004 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.715007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.715050 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.715129 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.716263 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.716980 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.718145 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.718479 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.718552 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.728661 (+ 10109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.728665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.728667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.728692 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.730415 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.730925 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.730930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.730935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.731024 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.731335 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.731338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.731341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.731553 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.731657 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.732051 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.732172 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:07:30.735988 30044 master_runner.cc:263] No leader master found from master 4332eaf5038a4fb0a522b46bc570cac9
I20260619 11:07:30.736048 30044 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:07:30.737036 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:30.714990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.16.61.61:42555)
0619 11:07:30.715104 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:30.715108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:30.724242 (+ 9134us) server_negotiation.cc:407] Connection header received
0619 11:07:30.724406 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:30.724409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:30.724463 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:30.724519 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:30.725476 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.726292 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.727469 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:30.727862 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:30.727943 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:30.733473 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:30.733476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:30.733479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:30.733504 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:30.735153 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.735647 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.735651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.735652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.735758 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:30.736095 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:30.736099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:30.736100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:30.736308 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:30.736404 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:30.736771 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 11:07:30.736879 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 11:07:30.739104 30242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:07:30.739185 30242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:07:30.739439 30230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.739575 30230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:07:30.789203 30206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:30.789397 30206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.790446 30206 leader_election.cc:288] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e05e0f882194e59ad8e4707df0aaa0c (127.16.61.62:41537), 4332eaf5038a4fb0a522b46bc570cac9 (127.16.61.61:42555)
I20260619 11:07:30.791023 30011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30d221b0b76642c6a78b8672eb0de505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" is_pre_election: true
I20260619 11:07:30.791307 30011 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d221b0b76642c6a78b8672eb0de505 in term 0.
I20260619 11:07:30.792088 30136 leader_election.cc:302] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e05e0f882194e59ad8e4707df0aaa0c, 30d221b0b76642c6a78b8672eb0de505; no voters:
I20260619 11:07:30.792891 30086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30d221b0b76642c6a78b8672eb0de505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4332eaf5038a4fb0a522b46bc570cac9" is_pre_election: true
I20260619 11:07:30.793151 30086 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30d221b0b76642c6a78b8672eb0de505 in term 0.
I20260619 11:07:30.793437 30206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:07:30.793535 30206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:07:30.793576 30206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:30.794849 30206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.795156 30206 leader_election.cc:288] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [CANDIDATE]: Term 1 election: Requested vote from peers 1e05e0f882194e59ad8e4707df0aaa0c (127.16.61.62:41537), 4332eaf5038a4fb0a522b46bc570cac9 (127.16.61.61:42555)
I20260619 11:07:30.795609 30011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30d221b0b76642c6a78b8672eb0de505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e05e0f882194e59ad8e4707df0aaa0c"
I20260619 11:07:30.795745 30011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:30.796952 30011 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d221b0b76642c6a78b8672eb0de505 in term 1.
I20260619 11:07:30.797310 30136 leader_election.cc:302] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e05e0f882194e59ad8e4707df0aaa0c, 30d221b0b76642c6a78b8672eb0de505; no voters:
I20260619 11:07:30.797819 30086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "30d221b0b76642c6a78b8672eb0de505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4332eaf5038a4fb0a522b46bc570cac9"
I20260619 11:07:30.797935 30086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:30.799158 30086 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30d221b0b76642c6a78b8672eb0de505 in term 1.
I20260619 11:07:30.801236 30206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:30.801460 30206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [term 1 LEADER]: Becoming Leader. State: Replica: 30d221b0b76642c6a78b8672eb0de505, State: Running, Role: LEADER
I20260619 11:07:30.801779 30206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } }
I20260619 11:07:30.803557 30206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d221b0b76642c6a78b8672eb0de505. Latest consensus state: current_term: 1 leader_uuid: "30d221b0b76642c6a78b8672eb0de505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.803731 30206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:30.804134 30246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:07:30.805307 30246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:07:30.814100 30086 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [term 1 FOLLOWER]: Refusing update from remote peer 30d221b0b76642c6a78b8672eb0de505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:30.814767 30244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:07:30.818369 30230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d221b0b76642c6a78b8672eb0de505. Latest consensus state: current_term: 1 leader_uuid: "30d221b0b76642c6a78b8672eb0de505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.818490 30230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:07:30.819538 30011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [term 1 FOLLOWER]: Refusing update from remote peer 30d221b0b76642c6a78b8672eb0de505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:30.820225 30244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:07:30.825824 30205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d221b0b76642c6a78b8672eb0de505. Latest consensus state: current_term: 1 leader_uuid: "30d221b0b76642c6a78b8672eb0de505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.825930 30205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:07:30.831256 30244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30d221b0b76642c6a78b8672eb0de505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.831372 30244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:30.832479 30230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30d221b0b76642c6a78b8672eb0de505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.832578 30230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:07:30.836308 30247 mvcc.cc:205] Tried to move back new op lower bound from 7298528259318722560 to 7298528259287425024. Current Snapshot: MvccSnapshot[applied={T|T < 7298528259318722560}]
I20260619 11:07:30.837112 30248 mvcc.cc:205] Tried to move back new op lower bound from 7298528259318722560 to 7298528259287425024. Current Snapshot: MvccSnapshot[applied={T|T < 7298528259318722560}]
I20260619 11:07:30.840971 30206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30d221b0b76642c6a78b8672eb0de505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.841073 30206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:30.842108 30205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30d221b0b76642c6a78b8672eb0de505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e05e0f882194e59ad8e4707df0aaa0c" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41537 } } peers { permanent_uuid: "4332eaf5038a4fb0a522b46bc570cac9" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 42555 } } peers { permanent_uuid: "30d221b0b76642c6a78b8672eb0de505" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 38807 } } }
I20260619 11:07:30.842206 30205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:07:30.843936 30250 mvcc.cc:205] Tried to move back new op lower bound from 7298528259318722560 to 7298528259287425024. Current Snapshot: MvccSnapshot[applied={T|T < 7298528259318722560}]
I20260619 11:07:30.847276 30246 catalog_manager.cc:1347] Generated new cluster ID: fb1cb2a2ec74428b84c9761c6f04038e
I20260619 11:07:30.847342 30246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:07:30.886088 30246 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:07:30.887283 30246 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:07:30.907936 30246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d221b0b76642c6a78b8672eb0de505: Generated new TSK 0
I20260619 11:07:30.908804 30246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:07:31.724151 30222 catalog_manager.cc:1259] Loaded cluster ID: fb1cb2a2ec74428b84c9761c6f04038e
I20260619 11:07:31.724231 30222 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: loading cluster ID for follower catalog manager: success
I20260619 11:07:31.725961 30222 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: acquiring CA information for follower catalog manager: success
I20260619 11:07:31.727032 30222 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e05e0f882194e59ad8e4707df0aaa0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:07:31.740291 30242 catalog_manager.cc:1259] Loaded cluster ID: fb1cb2a2ec74428b84c9761c6f04038e
I20260619 11:07:31.740353 30242 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9: loading cluster ID for follower catalog manager: success
I20260619 11:07:31.741906 30242 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9: acquiring CA information for follower catalog manager: success
I20260619 11:07:31.743016 30242 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4332eaf5038a4fb0a522b46bc570cac9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:07:31.964808 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 29969
I20260619 11:07:31.971060 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30044
I20260619 11:07:31.976619 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30119
2026-06-19T11:07:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3062 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:07:32 dist-test-slave-8vcz krb5kdc[30260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:07:32 dist-test-slave-8vcz krb5kdc[30260](info): set up 2 sockets
Jun 19 11:07:32 dist-test-slave-8vcz krb5kdc[30260](info): commencing operation
krb5kdc: starting...
W20260619 11:07:33.011966 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:07:33 dist-test-slave-8vcz krb5kdc[30260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:07:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:07:33.125145 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35273
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:35273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:33.228643 30273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:33.228921 30273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:33.228993 30273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:33.232928 30273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:33.233013 30273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:33.233052 30273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:33.233112 30273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:33.233151 30273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:33.238160 30273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:35273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:33.239151 30273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:33.240499 30273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:33.248076 30279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:33.248101 30281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:33.248106 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:33.248488 30273 server_base.cc:1034] running on GCE node
I20260619 11:07:33.248967 30273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:33.250037 30273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:33.251219 30273 hybrid_clock.cc:648] HybridClock initialized: now 1781867253251173 us; error 57 us; skew 500 ppm
Jun 19 11:07:33 dist-test-slave-8vcz krb5kdc[30260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867253, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:33.253870 30273 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:07:33.254619 30273 webserver.cc:460] Webserver started at http://127.16.61.62:46619/ using document root <none> and password file <none>
I20260619 11:07:33.255086 30273 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:33.255141 30273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:33.255286 30273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:33.257172 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "adbbebc6cbbe44c39a8247cf596399be"
format_stamp: "Formatted at 2026-06-19 11:07:33 on dist-test-slave-8vcz"
server_key: "9cfb55d429381620318f75adff676651"
server_key_iv: "9d59741d7d3c36aafc1b169a25a2e41f"
server_key_version: "encryptionkey@0"
I20260619 11:07:33.257647 30273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "adbbebc6cbbe44c39a8247cf596399be"
format_stamp: "Formatted at 2026-06-19 11:07:33 on dist-test-slave-8vcz"
server_key: "9cfb55d429381620318f75adff676651"
server_key_iv: "9d59741d7d3c36aafc1b169a25a2e41f"
server_key_version: "encryptionkey@0"
I20260619 11:07:33.261032 30273 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:07:33.263110 30288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:33.264297 30273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:33.264393 30273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "adbbebc6cbbe44c39a8247cf596399be"
format_stamp: "Formatted at 2026-06-19 11:07:33 on dist-test-slave-8vcz"
server_key: "9cfb55d429381620318f75adff676651"
server_key_iv: "9d59741d7d3c36aafc1b169a25a2e41f"
server_key_version: "encryptionkey@0"
I20260619 11:07:33.264474 30273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:33.301508 30273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:33.304633 30273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:33.304780 30273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:33.311340 30273 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:35273
I20260619 11:07:33.311407 30340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:35273 every 8 connection(s)
I20260619 11:07:33.312296 30273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:07:33.314849 30341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:33.319931 30341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be: Bootstrap starting.
I20260619 11:07:33.320710 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30273
I20260619 11:07:33.320856 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:07:33.321113 16628 external_mini_cluster.cc:1428] Setting key b6d17ffe03123c0a1ba55f87d54d4c7b
I20260619 11:07:33.322491 30341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:33.323173 30341 log.cc:824] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:33.325205 30341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be: No bootstrap required, opened a new log
I20260619 11:07:33.329403 30341 raft_consensus.cc:348] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbbebc6cbbe44c39a8247cf596399be" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35273 } }
I20260619 11:07:33.329622 30341 raft_consensus.cc:374] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:33.329699 30341 raft_consensus.cc:729] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbbebc6cbbe44c39a8247cf596399be, State: Initialized, Role: FOLLOWER
Jun 19 11:07:33 dist-test-slave-8vcz krb5kdc[30260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:33.330184 30341 consensus_queue.cc:260] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbbebc6cbbe44c39a8247cf596399be" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35273 } }
I20260619 11:07:33.330372 30341 raft_consensus.cc:388] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:07:33.330444 30341 raft_consensus.cc:482] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:07:33.330541 30341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:33.331586 30341 raft_consensus.cc:504] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbbebc6cbbe44c39a8247cf596399be" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35273 } }
I20260619 11:07:33.331940 30341 leader_election.cc:302] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adbbebc6cbbe44c39a8247cf596399be; no voters:
I20260619 11:07:33.332300 30341 leader_election.cc:288] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:07:33.332355 30346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:33.332527 30346 raft_consensus.cc:686] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [term 1 LEADER]: Becoming Leader. State: Replica: adbbebc6cbbe44c39a8247cf596399be, State: Running, Role: LEADER
I20260619 11:07:33.332849 30346 consensus_queue.cc:237] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbbebc6cbbe44c39a8247cf596399be" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35273 } }
I20260619 11:07:33.333441 30341 sys_catalog.cc:564] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:07:33.333976 30348 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adbbebc6cbbe44c39a8247cf596399be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbbebc6cbbe44c39a8247cf596399be" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35273 } } }
I20260619 11:07:33.334106 30348 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:33.334456 30347 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [sys.catalog]: SysCatalogTable state changed. Reason: New leader adbbebc6cbbe44c39a8247cf596399be. Latest consensus state: current_term: 1 leader_uuid: "adbbebc6cbbe44c39a8247cf596399be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbbebc6cbbe44c39a8247cf596399be" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35273 } } }
I20260619 11:07:33.334569 30347 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:33.336072 30344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:33.322118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.16.61.62:35273)
0619 11:07:33.322516 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:33.322531 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:33.322564 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:07:33.323178 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:33.323194 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:33.323617 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:33.323989 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:33.324778 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:33.325787 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:33.326961 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:33.327415 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:33.327528 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:33.330462 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:33.330468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:33.330483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:33.330521 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:33.332326 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:33.332830 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:33.332837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:33.332845 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:33.332980 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:33.333305 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:33.333308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:33.333309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:33.333764 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:33.334649 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:33.334791 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 11:07:33.335086 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":128,"threads_started":1}
W20260619 11:07:33.338001 30361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:07:33.338132 30361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:07:33.338209 30352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:07:33.339046 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:07:33.339049 30352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:07:33.343640 30352 catalog_manager.cc:1347] Generated new cluster ID: 4fb59da7129b4b67a57d8891f986715a
I20260619 11:07:33.343729 30352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:07:33.353485 30352 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:07:33.354310 30352 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:07:33.362738 30352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adbbebc6cbbe44c39a8247cf596399be: Generated new TSK 0
I20260619 11:07:33.363301 30352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:07:33.394618 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30273
2026-06-19T11:07:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1416 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:07:33 dist-test-slave-8vcz krb5kdc[30368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:07:33 dist-test-slave-8vcz krb5kdc[30368](info): set up 2 sockets
Jun 19 11:07:33 dist-test-slave-8vcz krb5kdc[30368](info): commencing operation
krb5kdc: starting...
W20260619 11:07:34.423695 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:07:34 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:07:34.538954 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:32969
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:34.642230 30381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:34.642458 30381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:34.642498 30381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:34.646293 30381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:34.646337 30381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:34.646355 30381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:34.646371 30381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:34.646385 30381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:34.651273 30381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:34.652155 30381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:34.653023 30381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:34.660833 30386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:34.661000 30381 server_base.cc:1034] running on GCE node
W20260619 11:07:34.661175 30387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:34.661197 30389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:34.661815 30381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:34.662792 30381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:34.663978 30381 hybrid_clock.cc:648] HybridClock initialized: now 1781867254663936 us; error 49 us; skew 500 ppm
Jun 19 11:07:34 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867254, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:34.666589 30381 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:07:34.667372 30381 webserver.cc:460] Webserver started at http://127.16.61.62:33905/ using document root <none> and password file <none>
I20260619 11:07:34.667984 30381 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:34.668071 30381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:34.668298 30381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:34.670174 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "cb6e3ef97f5946b78ae8d300bf4b8927"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1f9287957b38b67d13b460082280f2ec"
server_key_iv: "8df731d56cf42c2d3f5d44cd77f24f52"
server_key_version: "encryptionkey@0"
I20260619 11:07:34.670729 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "cb6e3ef97f5946b78ae8d300bf4b8927"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1f9287957b38b67d13b460082280f2ec"
server_key_iv: "8df731d56cf42c2d3f5d44cd77f24f52"
server_key_version: "encryptionkey@0"
I20260619 11:07:34.674191 30381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:07:34.676411 30396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:34.677604 30381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:07:34.677753 30381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "cb6e3ef97f5946b78ae8d300bf4b8927"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1f9287957b38b67d13b460082280f2ec"
server_key_iv: "8df731d56cf42c2d3f5d44cd77f24f52"
server_key_version: "encryptionkey@0"
I20260619 11:07:34.677860 30381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:34.699101 30381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:34.702224 30381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:34.702426 30381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:34.709162 30381 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:32969
I20260619 11:07:34.709439 30448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:32969 every 8 connection(s)
I20260619 11:07:34.710105 30381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:07:34.712885 30449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:34.714051 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30381
I20260619 11:07:34.714186 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:07:34.714485 16628 external_mini_cluster.cc:1428] Setting key 35b8adbf51129c57399e4a2208aad8c6
I20260619 11:07:34.719056 30449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Bootstrap starting.
I20260619 11:07:34.721545 30449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:34.722218 30449 log.cc:824] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:07:34 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:34.724298 30449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: No bootstrap required, opened a new log
I20260619 11:07:34.728554 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:34.715842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39512 (local address 127.16.61.62:32969)
0619 11:07:34.716224 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:34.716239 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:34.716273 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:07:34.716899 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:34.716915 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:34.717230 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:34.717503 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:34.718405 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:34.719447 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:34.720616 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:34.721061 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:34.721180 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:34.723754 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:34.723760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:34.723774 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:34.723803 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:34.725654 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:34.726053 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:34.726058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:34.726064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:34.726207 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:34.726455 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:34.726458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:34.726460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:34.726978 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:34.727146 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:34.727341 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 11:07:34.727619 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":116,"threads_started":1}
I20260619 11:07:34.728744 30449 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:34.728994 30449 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:34.729051 30449 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e3ef97f5946b78ae8d300bf4b8927, State: Initialized, Role: FOLLOWER
I20260619 11:07:34.729480 30449 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:34.729637 30449 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:07:34.729702 30449 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:07:34.729794 30449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:34.730674 30449 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:34.730981 30449 leader_election.cc:302] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb6e3ef97f5946b78ae8d300bf4b8927; no voters:
I20260619 11:07:34.731266 30449 leader_election.cc:288] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:07:34.731427 30454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:34.731644 30454 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 1 LEADER]: Becoming Leader. State: Replica: cb6e3ef97f5946b78ae8d300bf4b8927, State: Running, Role: LEADER
I20260619 11:07:34.732033 30454 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:34.732432 30449 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:07:34.733561 30455 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb6e3ef97f5946b78ae8d300bf4b8927. Latest consensus state: current_term: 1 leader_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } } }
I20260619 11:07:34.733688 30455 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:34.733985 30456 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } } }
I20260619 11:07:34.734066 30463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:07:34.734104 30456 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:34.737077 30463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:07:34.742100 30463 catalog_manager.cc:1347] Generated new cluster ID: 027ff8aac7844aae8782e9736991ba4c
I20260619 11:07:34.742213 30463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:07:34.765122 30463 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:07:34.766263 30463 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:07:34.773428 30463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Generated new TSK 0
I20260619 11:07:34.774024 30463 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:07:34.806649 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32969
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:34.915395 30475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:34.915622 30475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:34.915714 30475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:34.919539 30475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:34.919584 30475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:34.919642 30475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:07:34.924634 30475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:34.925453 30475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:34.926301 30475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:07:34.935570 30475 server_base.cc:1034] running on GCE node
W20260619 11:07:34.935529 30481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:34.935510 30480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:34.935513 30483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:34.936218 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:34.936735 30475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:34.937932 30475 hybrid_clock.cc:648] HybridClock initialized: now 1781867254937916 us; error 24 us; skew 500 ppm
Jun 19 11:07:34 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867254, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:34.940629 30475 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:07:34.941356 30475 webserver.cc:460] Webserver started at http://127.16.61.1:43411/ using document root <none> and password file <none>
I20260619 11:07:34.941828 30475 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:34.941883 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:34.942037 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:34.943912 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1ee766a41f8cdd346234445ead274a86"
server_key_iv: "381d8105d48a7fc33ce4baac988e3506"
server_key_version: "encryptionkey@0"
I20260619 11:07:34.944394 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1ee766a41f8cdd346234445ead274a86"
server_key_iv: "381d8105d48a7fc33ce4baac988e3506"
server_key_version: "encryptionkey@0"
I20260619 11:07:34.947789 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:07:34.950034 30490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:34.951193 30475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:34.951337 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1ee766a41f8cdd346234445ead274a86"
server_key_iv: "381d8105d48a7fc33ce4baac988e3506"
server_key_version: "encryptionkey@0"
I20260619 11:07:34.951448 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:34.970882 30475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:34.974098 30475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:34.974295 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:34.974864 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:34.975648 30475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:34.975737 30475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:34.975806 30475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:34.975849 30475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:34.984880 30475 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:40579
I20260619 11:07:34.984910 30603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:40579 every 8 connection(s)
I20260619 11:07:34.985751 30475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:07:34.992272 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30475
I20260619 11:07:34.992412 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:07:34.992749 16628 external_mini_cluster.cc:1428] Setting key 34cd4c8e35a6f71e481e6e74870d60ac
Jun 19 11:07:34 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867254, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:34.998835 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:34.987489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:33319 (local address 127.16.61.62:32969)
0619 11:07:34.987616 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:34.987620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:34.988522 (+ 902us) server_negotiation.cc:407] Connection header received
0619 11:07:34.989486 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:34.989490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:34.989555 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:34.989645 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:34.990746 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:34.991266 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:34.992491 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:34.992724 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:34.992836 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:34.995141 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:34.995148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:34.995154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:34.995185 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:34.996673 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:34.997268 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:34.997271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:34.997272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:34.997343 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:34.997736 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:34.997739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:34.997740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:34.997899 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:34.997975 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:34.998632 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 11:07:34.998724 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:07:35.000027 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:34.987772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:33319)
0619 11:07:34.988376 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:34.988413 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:34.989259 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:34.989819 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:34.989825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:34.990157 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:34.990605 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:34.990618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:34.991390 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:34.991393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:34.992357 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:34.992372 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:34.992870 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:34.992874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:34.993038 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:34.993573 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:34.993588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:34.994989 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:34.996786 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:34.996794 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:34.996809 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:34.997176 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:34.997431 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:34.997437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:34.997442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:34.997643 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:34.998020 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:34.998030 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:34.998344 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 11:07:34.998571 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 11:07:34.998915 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":107,"threads_started":1}
I20260619 11:07:35.001261 30604 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32969
I20260619 11:07:35.001498 30604 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:35.001961 30604 heartbeater.cc:511] Master 127.16.61.62:32969 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:07:35.003418 30413 ts_manager.cc:194] Registered new tserver with Master: 4bcc1a7d717d4ab6adfa6ff129e82f5f (127.16.61.1:40579)
I20260619 11:07:35.004667 30413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:33319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:07:35.020303 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32969
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:35.124382 30609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:35.124657 30609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:35.124730 30609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:35.128628 30609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:35.128708 30609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:35.128801 30609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:07:35.134179 30609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:35.135140 30609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:35.136145 30609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:35.145262 30615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:35.145269 30614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:35.145265 30617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:35.145573 30609 server_base.cc:1034] running on GCE node
I20260619 11:07:35.145963 30609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:35.146682 30609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:35.147835 30609 hybrid_clock.cc:648] HybridClock initialized: now 1781867255147807 us; error 39 us; skew 500 ppm
Jun 19 11:07:35 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:35.150513 30609 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:07:35.151324 30609 webserver.cc:460] Webserver started at http://127.16.61.2:41845/ using document root <none> and password file <none>
I20260619 11:07:35.151922 30609 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:35.152017 30609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:35.152240 30609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:35.154215 30609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "5b937b9050d2a13d291e1122c921ab0d"
server_key_iv: "df3a4fe137527e3beebafe114acd5b32"
server_key_version: "encryptionkey@0"
I20260619 11:07:35.154755 30609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "5b937b9050d2a13d291e1122c921ab0d"
server_key_iv: "df3a4fe137527e3beebafe114acd5b32"
server_key_version: "encryptionkey@0"
I20260619 11:07:35.158385 30609 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:07:35.160722 30624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:35.161859 30609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:07:35.162002 30609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "5b937b9050d2a13d291e1122c921ab0d"
server_key_iv: "df3a4fe137527e3beebafe114acd5b32"
server_key_version: "encryptionkey@0"
I20260619 11:07:35.162109 30609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:35.170701 30609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:35.173779 30609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:35.173977 30609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:35.174508 30609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:35.175279 30609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:35.175348 30609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:35.175414 30609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:35.175469 30609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:35.184170 30609 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:38881
I20260619 11:07:35.184191 30737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:38881 every 8 connection(s)
I20260619 11:07:35.185006 30609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:07:35.185791 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30609
I20260619 11:07:35.185926 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:07:35.186234 16628 external_mini_cluster.cc:1428] Setting key 71b951ba7af88b1703343b08e30b8127
Jun 19 11:07:35 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:07:35.199803 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.186803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:38747 (local address 127.16.61.62:32969)
0619 11:07:35.186923 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.186927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.187928 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 11:07:35.189315 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.189318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.189362 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.189431 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:35.190778 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.191278 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.193101 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.193313 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.193394 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.196164 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:35.196168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:35.196170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:35.196193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:35.197667 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.198199 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.198203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.198204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.198276 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.198616 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.198619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.198623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.198798 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:35.198905 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.199508 (+ 603us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.199614 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:07:35.200901 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.187152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:38747)
0619 11:07:35.187793 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.187833 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.189079 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.189604 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.189614 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.190118 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:35.190657 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.190671 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.191404 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.191410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.192975 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.192988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.193483 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.193491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.193725 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.194333 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:35.194355 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:35.195985 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:35.197804 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:35.197809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:35.197824 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:35.198082 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:35.198367 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:35.198372 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:35.198374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:35.198510 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:35.198902 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:35.198909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:35.199204 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.199426 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.199820 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":108,"threads_started":1}
I20260619 11:07:35.202116 30738 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32969
I20260619 11:07:35.202344 30738 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:35.202819 30738 heartbeater.cc:511] Master 127.16.61.62:32969 requested a full tablet report, sending...
I20260619 11:07:35.203891 30413 ts_manager.cc:194] Registered new tserver with Master: cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881)
I20260619 11:07:35.204444 30413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:38747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:07:35.215061 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32969
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:35.323274 30743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:35.323573 30743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:35.323654 30743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:35.327561 30743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:35.327641 30743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:35.327751 30743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:07:35.332876 30743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:35.333894 30743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:35.334859 30743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:35.344066 30749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:35.344072 30748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:35.344063 30751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:35.344383 30743 server_base.cc:1034] running on GCE node
I20260619 11:07:35.344755 30743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:35.345283 30743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:35.346450 30743 hybrid_clock.cc:648] HybridClock initialized: now 1781867255346437 us; error 23 us; skew 500 ppm
Jun 19 11:07:35 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:35.354132 30743 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:07:35.354981 30743 webserver.cc:460] Webserver started at http://127.16.61.3:46223/ using document root <none> and password file <none>
I20260619 11:07:35.355540 30743 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:35.355652 30743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:35.355887 30743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:35.357715 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "255276f3c83b4afdad2fefb4b89658e2"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "23c997fae26e8f786b3b9e8e5e9c580e"
server_key_iv: "7153b92ad65a73cb61d97943afc37503"
server_key_version: "encryptionkey@0"
I20260619 11:07:35.358191 30743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "255276f3c83b4afdad2fefb4b89658e2"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "23c997fae26e8f786b3b9e8e5e9c580e"
server_key_iv: "7153b92ad65a73cb61d97943afc37503"
server_key_version: "encryptionkey@0"
I20260619 11:07:35.361681 30743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:07:35.363864 30758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:35.365046 30743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:07:35.365159 30743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "255276f3c83b4afdad2fefb4b89658e2"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "23c997fae26e8f786b3b9e8e5e9c580e"
server_key_iv: "7153b92ad65a73cb61d97943afc37503"
server_key_version: "encryptionkey@0"
I20260619 11:07:35.365255 30743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:35.383315 30743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:35.386183 30743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:35.386371 30743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:35.386912 30743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:35.387712 30743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:35.387765 30743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:35.387825 30743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:35.387862 30743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:35.396200 30743 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:42509
I20260619 11:07:35.396226 30871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:42509 every 8 connection(s)
I20260619 11:07:35.396991 30743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:07:35.400727 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30743
I20260619 11:07:35.400849 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:07:35.401154 16628 external_mini_cluster.cc:1428] Setting key 09e3bdd0c844a5524111b4a474b67224
Jun 19 11:07:35 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:35.410076 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.398677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:46673 (local address 127.16.61.62:32969)
0619 11:07:35.398807 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.398810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.399583 (+ 773us) server_negotiation.cc:407] Connection header received
0619 11:07:35.401087 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.401090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.401136 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.401233 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:35.402676 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.403340 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.404473 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.404680 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.404760 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.406900 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:35.406903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:35.406905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:35.406929 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:35.408396 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.408833 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.408835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.408837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.408907 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.409184 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.409187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.409188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.409347 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:35.409443 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.409868 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.409964 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:07:35.410766 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.399017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:46673)
0619 11:07:35.399430 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.399468 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.400867 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.401391 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.401399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.401908 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:35.402540 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.402561 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.403447 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.403450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.404369 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.404378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.404802 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.404804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.404934 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.405421 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:35.405436 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:35.406763 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:35.408505 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:35.408509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:35.408519 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:35.408752 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:35.408984 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:35.408986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:35.408987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:35.409107 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:35.409430 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:35.409434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:35.409634 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.409777 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.409990 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260619 11:07:35.411844 30872 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32969
I20260619 11:07:35.412075 30872 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:35.412521 30872 heartbeater.cc:511] Master 127.16.61.62:32969 requested a full tablet report, sending...
I20260619 11:07:35.413446 30413 ts_manager.cc:194] Registered new tserver with Master: 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509)
I20260619 11:07:35.414129 30413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:46673
I20260619 11:07:35.414863 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:07:35.423122 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.416719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.16.61.62:32969)
0619 11:07:35.416853 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.416857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.416984 (+ 127us) server_negotiation.cc:407] Connection header received
0619 11:07:35.417062 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.417064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.417098 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.417155 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:35.417785 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.418263 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.419025 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.419222 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.419295 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.420555 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:35.420558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:35.420559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:35.420579 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:35.421743 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.422095 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.422097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.422099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.422165 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.422382 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.422384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.422385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.422585 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:35.422685 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.422855 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.422936 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:07:35.425725 30412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:07:35.428563 30412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:07:35.442369 30887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.437113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.16.61.2:38881)
0619 11:07:35.438383 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.438391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.438408 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:07:35.438463 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.438466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.438596 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.438738 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:35.438916 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.439798 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.441219 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.441742 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.441859 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.442003 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.442137 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.442227 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":138,"threads_started":1}
I20260619 11:07:35.442373 30885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.436630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:38881 (local address 127.0.0.1:38892)
0619 11:07:35.436845 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.436858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.436947 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.438722 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.438725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.438741 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:35.438825 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.438830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.439928 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.439931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.441098 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.441105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.441870 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.441874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.441974 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.442009 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.442111 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.442244 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":30,"thread_start_us":104,"threads_started":1}
I20260619 11:07:35.442401 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.436326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42509 (local address 127.0.0.1:39410)
0619 11:07:35.436627 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.436679 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.436793 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.437757 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.437765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.437785 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:35.437942 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.437953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.439462 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.439466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.440859 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.440868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.441936 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.441940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.442032 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.442054 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.442106 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.442241 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":88,"thread_start_us":81,"threads_started":1}
I20260619 11:07:35.442662 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.437229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.16.61.1:40579)
0619 11:07:35.437644 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.437659 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.437682 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:07:35.437755 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.437760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.437938 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.438096 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:35.438316 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.439409 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.441061 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.441804 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.441912 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.442054 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.442436 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.442527 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":108,"threads_started":1}
I20260619 11:07:35.442834 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.437077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:40579 (local address 127.0.0.1:35260)
0619 11:07:35.437554 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.437568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.437648 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.438074 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.438077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.438094 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:35.438186 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.438192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.439516 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.439520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.440957 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.440967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.442113 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.442117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.442198 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.442216 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.442289 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.442348 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":81,"thread_start_us":80,"threads_started":1}
I20260619 11:07:35.442880 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.436444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.16.61.3:42509)
0619 11:07:35.437283 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.437296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.437319 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:07:35.437388 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.437395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.437587 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.438079 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:35.438182 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.439324 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.440977 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.441796 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.442467 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.442596 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.442641 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.442734 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":128,"threads_started":1}
I20260619 11:07:35.445333 30806 tablet_service.cc:1467] Processing CreateTablet for tablet 3a11b56143664cab9d41763cbcb3bb89 (DEFAULT_TABLE table=test-table [id=019b0b6e6de549a4a99820d9f6f3fea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:35.445334 30538 tablet_service.cc:1467] Processing CreateTablet for tablet 3a11b56143664cab9d41763cbcb3bb89 (DEFAULT_TABLE table=test-table [id=019b0b6e6de549a4a99820d9f6f3fea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:35.446195 30538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a11b56143664cab9d41763cbcb3bb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:35.446203 30806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a11b56143664cab9d41763cbcb3bb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:35.451571 30889 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Bootstrap starting.
I20260619 11:07:35.451572 30672 tablet_service.cc:1467] Processing CreateTablet for tablet 3a11b56143664cab9d41763cbcb3bb89 (DEFAULT_TABLE table=test-table [id=019b0b6e6de549a4a99820d9f6f3fea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:35.452553 30672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a11b56143664cab9d41763cbcb3bb89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:35.453154 30890 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: Bootstrap starting.
I20260619 11:07:35.454419 30889 tablet_bootstrap.cc:654] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:35.455567 30889 log.cc:824] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:35.455590 30890 tablet_bootstrap.cc:654] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:35.456415 30890 log.cc:824] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:35.457408 30889 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: No bootstrap required, opened a new log
I20260619 11:07:35.457603 30889 ts_tablet_manager.cc:1397] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 11:07:35.458207 30890 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: No bootstrap required, opened a new log
I20260619 11:07:35.458364 30890 ts_tablet_manager.cc:1397] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:07:35.459280 30893 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Bootstrap starting.
I20260619 11:07:35.461441 30890 raft_consensus.cc:348] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.461395 30889 raft_consensus.cc:348] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.461661 30890 raft_consensus.cc:374] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:35.461666 30889 raft_consensus.cc:374] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:35.461719 30890 raft_consensus.cc:729] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255276f3c83b4afdad2fefb4b89658e2, State: Initialized, Role: FOLLOWER
I20260619 11:07:35.461719 30889 raft_consensus.cc:729] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcc1a7d717d4ab6adfa6ff129e82f5f, State: Initialized, Role: FOLLOWER
I20260619 11:07:35.461819 30893 tablet_bootstrap.cc:654] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:35.462142 30890 consensus_queue.cc:260] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.462188 30889 consensus_queue.cc:260] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.462631 30893 log.cc:824] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:35.462771 30872 heartbeater.cc:503] Master 127.16.61.62:32969 was elected leader, sending a full tablet report...
I20260619 11:07:35.463114 30889 ts_tablet_manager.cc:1428] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:07:35.463465 30604 heartbeater.cc:503] Master 127.16.61.62:32969 was elected leader, sending a full tablet report...
I20260619 11:07:35.464051 30890 ts_tablet_manager.cc:1428] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:07:35.464648 30893 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: No bootstrap required, opened a new log
I20260619 11:07:35.465126 30893 ts_tablet_manager.cc:1397] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:07:35.468609 30893 raft_consensus.cc:348] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.468825 30893 raft_consensus.cc:374] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:35.468883 30893 raft_consensus.cc:729] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2d80f35b6d4cb2b0782cf43d7bbbf0, State: Initialized, Role: FOLLOWER
I20260619 11:07:35.469223 30893 consensus_queue.cc:260] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.469790 30738 heartbeater.cc:503] Master 127.16.61.62:32969 was elected leader, sending a full tablet report...
I20260619 11:07:35.470108 30893 ts_tablet_manager.cc:1428] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260619 11:07:35.486836 30605 tablet.cc:2369] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:07:35.488587 30895 raft_consensus.cc:482] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:35.488724 30895 raft_consensus.cc:504] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.489727 30895 leader_election.cc:288] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509), cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881)
I20260619 11:07:35.494371 30887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.490171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:34959 (local address 127.16.61.2:38881)
0619 11:07:35.490284 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.490288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.490432 (+ 144us) server_negotiation.cc:407] Connection header received
0619 11:07:35.490540 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.490542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.490577 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.490635 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:35.490900 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.491591 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.493137 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.493734 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.493843 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.493911 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.494190 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.494254 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:07:35.494371 30884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.489987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:53479 (local address 127.16.61.3:42509)
0619 11:07:35.490154 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.490158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.490173 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:07:35.490221 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.490234 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.490287 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.490345 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:35.490721 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.491503 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.493137 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.493745 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.493843 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.493902 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.494190 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.494254 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 11:07:35.494405 30606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.489873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42509 (local address 127.16.61.1:53479)
0619 11:07:35.490015 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.490031 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.490117 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.490405 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.490408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.490444 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:35.490587 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.490592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.491613 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.491615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.493013 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.493020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.493873 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.493877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.494028 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.494076 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.494157 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.494249 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":104,"spinlock_wait_cycles":5504}
I20260619 11:07:35.494413 30898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.489993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:38881 (local address 127.16.61.1:34959)
0619 11:07:35.490316 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.490329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.490446 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.490678 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.490681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.490699 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:35.490795 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.490799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.491716 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.491719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.493009 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.493020 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.493881 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.493884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.494009 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.494083 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.494157 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.494248 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":131,"spinlock_wait_cycles":22272,"thread_start_us":122,"threads_started":1}
I20260619 11:07:35.495123 30692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a11b56143664cab9d41763cbcb3bb89" candidate_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" is_pre_election: true
I20260619 11:07:35.495123 30826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a11b56143664cab9d41763cbcb3bb89" candidate_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "255276f3c83b4afdad2fefb4b89658e2" is_pre_election: true
I20260619 11:07:35.495393 30826 raft_consensus.cc:2454] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcc1a7d717d4ab6adfa6ff129e82f5f in term 0.
I20260619 11:07:35.495422 30692 raft_consensus.cc:2454] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcc1a7d717d4ab6adfa6ff129e82f5f in term 0.
I20260619 11:07:35.495885 30493 leader_election.cc:302] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 255276f3c83b4afdad2fefb4b89658e2, 4bcc1a7d717d4ab6adfa6ff129e82f5f, cf2d80f35b6d4cb2b0782cf43d7bbbf0; no voters:
I20260619 11:07:35.496176 30895 raft_consensus.cc:2781] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:07:35.496279 30895 raft_consensus.cc:482] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:07:35.496340 30895 raft_consensus.cc:3037] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:35.497654 30895 raft_consensus.cc:504] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.498010 30895 leader_election.cc:288] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 1 election: Requested vote from peers 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509), cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881)
I20260619 11:07:35.498319 30826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a11b56143664cab9d41763cbcb3bb89" candidate_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "255276f3c83b4afdad2fefb4b89658e2"
I20260619 11:07:35.498319 30692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a11b56143664cab9d41763cbcb3bb89" candidate_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0"
I20260619 11:07:35.498431 30692 raft_consensus.cc:3037] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:35.498438 30826 raft_consensus.cc:3037] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:35.499631 30826 raft_consensus.cc:2454] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcc1a7d717d4ab6adfa6ff129e82f5f in term 1.
I20260619 11:07:35.499631 30692 raft_consensus.cc:2454] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcc1a7d717d4ab6adfa6ff129e82f5f in term 1.
I20260619 11:07:35.499930 30492 leader_election.cc:302] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 255276f3c83b4afdad2fefb4b89658e2, 4bcc1a7d717d4ab6adfa6ff129e82f5f; no voters:
I20260619 11:07:35.500085 30895 raft_consensus.cc:2781] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:35.500308 30895 raft_consensus.cc:686] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 LEADER]: Becoming Leader. State: Replica: 4bcc1a7d717d4ab6adfa6ff129e82f5f, State: Running, Role: LEADER
I20260619 11:07:35.500607 30895 consensus_queue.cc:237] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.503408 30413 catalog_manager.cc:5495] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bcc1a7d717d4ab6adfa6ff129e82f5f (127.16.61.1). New cstate: current_term: 1 leader_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20260619 11:07:35.520632 30888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.517544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.16.61.1:40579)
0619 11:07:35.517653 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.517656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.517668 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:07:35.517794 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.517796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.517841 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.517921 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:07:35.518126 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.518630 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.519418 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.519625 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.519770 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.519991 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:07:35.520337 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:07:35.520423 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.520483 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.520541 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:07:35.525753 30826 raft_consensus.cc:1261] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 FOLLOWER]: Refusing update from remote peer 4bcc1a7d717d4ab6adfa6ff129e82f5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:35.525753 30692 raft_consensus.cc:1261] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 1 FOLLOWER]: Refusing update from remote peer 4bcc1a7d717d4ab6adfa6ff129e82f5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:35.526449 30895 consensus_queue.cc:1035] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:07:35.526599 30899 consensus_queue.cc:1035] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:07:35.535471 30903 mvcc.cc:205] Tried to move back new op lower bound from 7298528278626918400 to 7298528278532751360. Current Snapshot: MvccSnapshot[applied={T|T < 7298528278626918400}]
I20260619 11:07:35.540875 30412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:07:35.541004 30412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:07:35.543232 30412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Sending DeleteTablet for 3 replicas of tablet 3a11b56143664cab9d41763cbcb3bb89
I20260619 11:07:35.544005 30672 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a11b56143664cab9d41763cbcb3bb89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:35 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:38892
I20260619 11:07:35.544029 30538 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a11b56143664cab9d41763cbcb3bb89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:35 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:35260
I20260619 11:07:35.544461 30912 tablet_replica.cc:330] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: stopping tablet replica
I20260619 11:07:35.544503 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30475
I20260619 11:07:35.544483 30806 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a11b56143664cab9d41763cbcb3bb89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:35 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:39410
I20260619 11:07:35.544881 30912 raft_consensus.cc:2229] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:07:35.545251 30912 raft_consensus.cc:2258] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:35.546208 30914 tablet_replica.cc:330] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: stopping tablet replica
I20260619 11:07:35.546460 30914 raft_consensus.cc:2229] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:07:35.546717 30914 raft_consensus.cc:2258] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:35.546751 30912 ts_tablet_manager.cc:1905] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:07:35.547968 30914 ts_tablet_manager.cc:1905] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:07:35.549458 30912 ts_tablet_manager.cc:1918] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:07:35.549568 30912 log.cc:1196] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/3a11b56143664cab9d41763cbcb3bb89
I20260619 11:07:35.549926 30912 ts_tablet_manager.cc:1939] T 3a11b56143664cab9d41763cbcb3bb89 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Deleting consensus metadata
I20260619 11:07:35.550576 30914 ts_tablet_manager.cc:1918] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:07:35.550674 30914 log.cc:1196] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/3a11b56143664cab9d41763cbcb3bb89
I20260619 11:07:35.550990 30914 ts_tablet_manager.cc:1939] T 3a11b56143664cab9d41763cbcb3bb89 P 255276f3c83b4afdad2fefb4b89658e2: Deleting consensus metadata
I20260619 11:07:35.551204 30399 catalog_manager.cc:4841] TS cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881): tablet 3a11b56143664cab9d41763cbcb3bb89 (table test-table [id=019b0b6e6de549a4a99820d9f6f3fea7]) successfully deleted
W20260619 11:07:35.552004 30398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:40579 (error 108)
I20260619 11:07:35.552145 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30609
I20260619 11:07:35.552461 30398 catalog_manager.cc:4841] TS 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509): tablet 3a11b56143664cab9d41763cbcb3bb89 (table test-table [id=019b0b6e6de549a4a99820d9f6f3fea7]) successfully deleted
I20260619 11:07:35.553758 30886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.553495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:40579 (local address 127.0.0.1:35278)
0619 11:07:35.553619 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.553691 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:40579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 11:07:35.553891 30398 catalog_manager.cc:4584] TS 4bcc1a7d717d4ab6adfa6ff129e82f5f (127.16.61.1:40579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a11b56143664cab9d41763cbcb3bb89: Network error: Client connection negotiation failed: client connection to 127.16.61.1:40579: connect: Connection refused (error 111)
I20260619 11:07:35.558506 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30743
I20260619 11:07:35.563792 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30381
I20260619 11:07:35.569976 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:32969
--webserver_interface=127.16.61.62
--webserver_port=33905
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:35.674850 30916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:35.675118 30916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:35.675189 30916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:35.679132 30916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:35.679217 30916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:35.679257 30916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:35.679313 30916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:35.679352 30916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:35.684414 30916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:35.685359 30916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:35.686317 30916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:35.698266 30923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:35.698333 30925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:35.698333 30922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:35.698530 30916 server_base.cc:1034] running on GCE node
I20260619 11:07:35.699121 30916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:35.700204 30916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:35.701362 30916 hybrid_clock.cc:648] HybridClock initialized: now 1781867255701349 us; error 23 us; skew 500 ppm
Jun 19 11:07:35 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:35.703948 30916 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:07:35.704713 30916 webserver.cc:460] Webserver started at http://127.16.61.62:33905/ using document root <none> and password file <none>
I20260619 11:07:35.705268 30916 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:35.705335 30916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:35.708873 30916 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:07:35.710799 30932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:35.712167 30916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:07:35.712289 30916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "cb6e3ef97f5946b78ae8d300bf4b8927"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1f9287957b38b67d13b460082280f2ec"
server_key_iv: "8df731d56cf42c2d3f5d44cd77f24f52"
server_key_version: "encryptionkey@0"
I20260619 11:07:35.712844 30916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:35.732842 30916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:35.735457 30916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:35.735658 30916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:35.742558 30916 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:32969
I20260619 11:07:35.742944 30984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:32969 every 8 connection(s)
I20260619 11:07:35.743650 30916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:07:35.745464 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30916
I20260619 11:07:35.745885 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:40579
--local_ip_for_outbound_sockets=127.16.61.1
--tserver_master_addrs=127.16.61.62:32969
--webserver_port=43411
--webserver_interface=127.16.61.1
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:07:35.748461 30985 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:07:35.750120 30985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Bootstrap starting.
I20260619 11:07:35.766475 30985 log.cc:824] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:35.774610 30985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:07:35.774979 30985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Bootstrap complete.
I20260619 11:07:35.778892 30985 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:35.779284 30985 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e3ef97f5946b78ae8d300bf4b8927, State: Initialized, Role: FOLLOWER
I20260619 11:07:35.779794 30985 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:35.779950 30985 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:07:35.780025 30985 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:07:35.780135 30985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:07:35.781936 30985 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:35.782256 30985 leader_election.cc:302] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb6e3ef97f5946b78ae8d300bf4b8927; no voters:
I20260619 11:07:35.782579 30985 leader_election.cc:288] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:07:35.782658 30989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:07:35.782845 30989 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 2 LEADER]: Becoming Leader. State: Replica: cb6e3ef97f5946b78ae8d300bf4b8927, State: Running, Role: LEADER
I20260619 11:07:35.783123 30989 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:35.783776 30985 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:07:35.784386 30990 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } } }
I20260619 11:07:35.784499 30990 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:35.785231 30996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:07:35.785439 30991 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb6e3ef97f5946b78ae8d300bf4b8927. Latest consensus state: current_term: 2 leader_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } } }
I20260619 11:07:35.785612 30991 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:35.789857 30996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:07:35.790666 30996 catalog_manager.cc:1259] Loaded cluster ID: 027ff8aac7844aae8782e9736991ba4c
I20260619 11:07:35.790723 30996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:07:35.792107 30996 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 11:07:35.796983 30996 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 11:07:35.801996 30996 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:07:35.802675 30996 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Loaded TSK: 0
I20260619 11:07:35.803823 30996 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:07:35.856860 30987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:35.857116 30987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:35.857198 30987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:35.861219 30987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:35.861287 30987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:35.861389 30987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:07:35.866458 30987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:40579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=43411
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:35.867410 30987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:35.868395 30987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:35.877372 31012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:35.877408 31013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:35.877696 31015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:35.877696 30987 server_base.cc:1034] running on GCE node
I20260619 11:07:35.878139 30987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:35.878839 30987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:35.880040 30987 hybrid_clock.cc:648] HybridClock initialized: now 1781867255880015 us; error 31 us; skew 500 ppm
Jun 19 11:07:35 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:35.883323 30987 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:07:35.884471 30987 webserver.cc:460] Webserver started at http://127.16.61.1:43411/ using document root <none> and password file <none>
I20260619 11:07:35.885161 30987 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:35.885262 30987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:35.890038 30987 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 11:07:35.892668 31022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:35.894110 30987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:07:35.894227 30987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1ee766a41f8cdd346234445ead274a86"
server_key_iv: "381d8105d48a7fc33ce4baac988e3506"
server_key_version: "encryptionkey@0"
I20260619 11:07:35.894886 30987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:35.911996 30987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:35.915287 30987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:35.915535 30987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:35.916394 30987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:35.917634 31029 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:07:35.921573 30987 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:07:35.921635 30987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 11:07:35.921687 30987 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:07:35.923309 30987 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:07:35.923364 30987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 11:07:35.923611 31029 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Bootstrap starting.
I20260619 11:07:35.933665 30987 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:40579
I20260619 11:07:35.933835 31136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:40579 every 8 connection(s)
I20260619 11:07:35.934900 30987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:07:35.942212 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 30987
I20260619 11:07:35.942675 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:38881
--local_ip_for_outbound_sockets=127.16.61.2
--tserver_master_addrs=127.16.61.62:32969
--webserver_port=41845
--webserver_interface=127.16.61.2
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:07:35.944500 31029 log.cc:824] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:07:35 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:35.952808 31029 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:07:35.953260 31029 tablet_bootstrap.cc:492] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Bootstrap complete.
I20260619 11:07:35.954474 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.937183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:41711 (local address 127.16.61.62:32969)
0619 11:07:35.937623 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.937644 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.938077 (+ 433us) server_negotiation.cc:407] Connection header received
0619 11:07:35.939556 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.939575 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.940014 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.940368 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:35.941350 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.942936 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.944594 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.945195 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.945366 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.948479 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:35.948485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:35.948500 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:35.948529 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:35.950645 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.951338 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.951344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.951349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.951492 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.951973 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.951976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.951977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.952399 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:35.952531 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.953258 (+ 727us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.953617 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":99,"threads_started":1}
I20260619 11:07:35.954494 31140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.937225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:41711)
0619 11:07:35.937914 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.937964 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.939013 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.940403 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.940409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.940747 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:35.941171 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.941183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.943115 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.943122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.944467 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.944477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.945351 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.945355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.945561 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.946248 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:35.946268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:35.948262 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:35.950781 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:35.950791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:35.950806 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:35.951205 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:35.951603 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:35.951609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:35.951614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:35.951848 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:35.952532 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:35.952542 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:35.952872 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.953095 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.953464 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":138,"threads_started":1}
I20260619 11:07:35.954627 31029 ts_tablet_manager.cc:1397] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.007s
I20260619 11:07:35.957204 31137 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32969
I20260619 11:07:35.957429 31137 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:35.957158 31029 raft_consensus.cc:348] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.957669 31029 raft_consensus.cc:729] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcc1a7d717d4ab6adfa6ff129e82f5f, State: Initialized, Role: FOLLOWER
I20260619 11:07:35.957959 31137 heartbeater.cc:511] Master 127.16.61.62:32969 requested a full tablet report, sending...
I20260619 11:07:35.958168 31029 consensus_queue.cc:260] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } } peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } }
I20260619 11:07:35.959131 31029 ts_tablet_manager.cc:1428] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260619 11:07:35.960721 30947 ts_manager.cc:194] Registered new tserver with Master: 4bcc1a7d717d4ab6adfa6ff129e82f5f (127.16.61.1:40579)
I20260619 11:07:35.964625 30947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:41711
I20260619 11:07:35.966781 31137 heartbeater.cc:503] Master 127.16.61.62:32969 was elected leader, sending a full tablet report...
Jun 19 11:07:35 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:07:35.974606 31144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.963951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:40579 (local address 127.0.0.1:35284)
0619 11:07:35.964446 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:35.964490 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:35.964616 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:35.965109 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:35.965115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:35.965592 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:35.965722 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.965730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.967174 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.967178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.968163 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:35.968174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.968947 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.968950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:35.969030 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.969529 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:35.969541 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:35.970999 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:35.972970 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:35.972977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:35.972981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:35.973186 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:35.973502 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:35.973504 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:35.973506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:35.973590 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:35.974119 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:35.974123 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:35.974230 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 11:07:35.974349 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 11:07:35.974487 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":100,"threads_started":1}
I20260619 11:07:35.974751 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:35.963990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.16.61.1:40579)
0619 11:07:35.964654 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:35.964668 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:35.964692 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:07:35.964798 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:35.964805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:35.964964 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:35.965108 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:35.965845 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.967043 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.968300 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:35.968800 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:35.968944 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:35.971135 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:35.971144 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:35.971155 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:35.971179 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:35.972862 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.973290 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.973296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.973298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.973386 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:35.973684 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:35.973688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:35.973689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:35.973991 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:35.974123 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:35.974446 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 11:07:35.974597 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":565,"thread_start_us":91,"threads_started":1}
I20260619 11:07:35.975623 31064 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a11b56143664cab9d41763cbcb3bb89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:35 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:35284
I20260619 11:07:35.976481 31146 tablet_replica.cc:330] stopping tablet replica
I20260619 11:07:35.976778 31146 raft_consensus.cc:2229] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:07:35.977016 31146 raft_consensus.cc:2258] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:35.977974 31146 ts_tablet_manager.cc:1905] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:07:35.982098 31146 ts_tablet_manager.cc:1918] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:07:35.982201 31146 log.cc:1196] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/3a11b56143664cab9d41763cbcb3bb89
I20260619 11:07:35.982488 31146 ts_tablet_manager.cc:1939] T 3a11b56143664cab9d41763cbcb3bb89 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Deleting consensus metadata
I20260619 11:07:35.984433 30934 catalog_manager.cc:4841] TS 4bcc1a7d717d4ab6adfa6ff129e82f5f (127.16.61.1:40579): tablet 3a11b56143664cab9d41763cbcb3bb89 (table test-table [id=019b0b6e6de549a4a99820d9f6f3fea7]) successfully deleted
W20260619 11:07:36.061183 31141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:36.061414 31141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:36.061455 31141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:36.065500 31141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:36.065554 31141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:36.065610 31141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:07:36.070590 31141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:38881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=41845
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:36.071517 31141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:36.072482 31141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:36.081475 31152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:36.081663 31141 server_base.cc:1034] running on GCE node
W20260619 11:07:36.081492 31151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:36.081569 31154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:36.082113 31141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:36.082702 31141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:36.083896 31141 hybrid_clock.cc:648] HybridClock initialized: now 1781867256083862 us; error 45 us; skew 500 ppm
Jun 19 11:07:36 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:36.086684 31141 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:07:36.087518 31141 webserver.cc:460] Webserver started at http://127.16.61.2:41845/ using document root <none> and password file <none>
I20260619 11:07:36.088065 31141 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:36.088135 31141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:36.091629 31141 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:07:36.093572 31161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:36.094823 31141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:36.094926 31141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "5b937b9050d2a13d291e1122c921ab0d"
server_key_iv: "df3a4fe137527e3beebafe114acd5b32"
server_key_version: "encryptionkey@0"
I20260619 11:07:36.095424 31141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:36.124843 31141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:36.128070 31141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:36.128219 31141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:36.128726 31141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:36.129463 31141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:36.129509 31141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:36.129547 31141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:36.129563 31141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:36.138696 31141 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:38881
I20260619 11:07:36.138725 31274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:38881 every 8 connection(s)
I20260619 11:07:36.139686 31141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:07:36.139987 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 31141
I20260619 11:07:36.140504 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:42509
--local_ip_for_outbound_sockets=127.16.61.3
--tserver_master_addrs=127.16.61.62:32969
--webserver_port=46223
--webserver_interface=127.16.61.3
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:07:36 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:36.153463 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.141677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:52881 (local address 127.16.61.62:32969)
0619 11:07:36.141844 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.141850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.142819 (+ 969us) server_negotiation.cc:407] Connection header received
0619 11:07:36.144231 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.144235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.144299 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.144400 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:36.145546 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.146076 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.147206 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.147438 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.147549 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.149945 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:36.149948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:36.149950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:36.149978 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:36.151398 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:36.151963 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:36.151968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:36.151981 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:36.152054 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:36.152405 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:36.152408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:36.152410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:36.152584 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:36.152690 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.153162 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.153288 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:07:36.154207 31278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.142149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:52881)
0619 11:07:36.142670 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:36.142711 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:36.144022 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:36.144578 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:36.144584 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:36.144928 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:36.145392 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.145405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.146201 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.146206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.147080 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.147088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.147590 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.147594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.147823 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.148401 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:36.148418 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:36.149775 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:36.151516 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:36.151522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:36.151535 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:36.151862 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:36.152145 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:36.152149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:36.152151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:36.152295 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:36.152687 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:36.152694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:36.152928 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 11:07:36.153107 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 11:07:36.153406 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":134,"threads_started":1}
I20260619 11:07:36.155489 31275 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32969
I20260619 11:07:36.155757 31275 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:36.156234 31275 heartbeater.cc:511] Master 127.16.61.62:32969 requested a full tablet report, sending...
I20260619 11:07:36.157271 30947 ts_manager.cc:194] Registered new tserver with Master: cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881)
I20260619 11:07:36.158006 30947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:52881
W20260619 11:07:36.251217 31277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:36.251475 31277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:36.251518 31277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:36.255353 31277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:36.255406 31277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:36.255468 31277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:07:36.260447 31277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:42509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=46223
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:36.261375 31277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:36.262308 31277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:36.271626 31284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:36.271716 31286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:36.271626 31283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:36.271853 31277 server_base.cc:1034] running on GCE node
I20260619 11:07:36.272403 31277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:36.273032 31277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:36.274216 31277 hybrid_clock.cc:648] HybridClock initialized: now 1781867256274197 us; error 28 us; skew 500 ppm
Jun 19 11:07:36 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:36.277074 31277 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:07:36.277853 31277 webserver.cc:460] Webserver started at http://127.16.61.3:46223/ using document root <none> and password file <none>
I20260619 11:07:36.278421 31277 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:36.278496 31277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:36.282269 31277 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:36.284545 31293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:36.285967 31277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:07:36.286065 31277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "255276f3c83b4afdad2fefb4b89658e2"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "23c997fae26e8f786b3b9e8e5e9c580e"
server_key_iv: "7153b92ad65a73cb61d97943afc37503"
server_key_version: "encryptionkey@0"
I20260619 11:07:36.286583 31277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:36.310617 31277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:36.313740 31277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:36.313890 31277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:36.314442 31277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:36.315323 31277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:36.315372 31277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:36.315412 31277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:36.315428 31277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:36.324242 31277 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:42509
I20260619 11:07:36.324271 31406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:42509 every 8 connection(s)
I20260619 11:07:36.325244 31277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:07:36.326994 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 31277
Jun 19 11:07:36 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:36.336433 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.327658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.16.61.62:32969)
0619 11:07:36.327903 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.327907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.328297 (+ 390us) server_negotiation.cc:407] Connection header received
0619 11:07:36.328346 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.328349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.328395 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.328493 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:36.329206 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.330012 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.330885 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.331155 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.331232 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.332664 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:36.332668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:36.332671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:36.332696 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:36.334499 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:36.334927 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:36.334930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:36.334931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:36.335004 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:36.335291 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:36.335293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:36.335295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:36.335442 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:36.335702 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.336060 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.336206 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260619 11:07:36.339869 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.326981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:60869 (local address 127.16.61.62:32969)
0619 11:07:36.327156 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.327160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.328156 (+ 996us) server_negotiation.cc:407] Connection header received
0619 11:07:36.329679 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.329681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.329720 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.329780 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:36.330979 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.331633 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.332935 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.333304 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.333389 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.336201 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:36.336204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:36.336207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:36.336227 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:36.337829 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:36.338410 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:36.338412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:36.338414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:36.338484 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:36.338826 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:36.338827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:36.338829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:36.338968 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:36.339082 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.339616 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.339758 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:07:36.340693 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.327406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:60869)
0619 11:07:36.327962 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:36.328032 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:36.329473 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:36.329951 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:36.329959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:36.330344 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:36.330815 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.330857 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.331787 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.331792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.332806 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.332818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.333459 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.333464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.333667 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.334152 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:36.334173 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:36.336030 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:36.337931 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:36.337935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:36.337948 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:36.338306 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:36.338593 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:36.338596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:36.338597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:36.338740 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:36.339154 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:36.339159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:36.339387 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 11:07:36.339555 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 11:07:36.339876 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":163,"threads_started":1}
I20260619 11:07:36.341789 31407 heartbeater.cc:347] Connected to a master server at 127.16.61.62:32969
I20260619 11:07:36.342052 31407 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:36.342535 31407 heartbeater.cc:511] Master 127.16.61.62:32969 requested a full tablet report, sending...
I20260619 11:07:36.343510 30947 ts_manager.cc:194] Registered new tserver with Master: 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509)
I20260619 11:07:36.344087 30947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:60869
I20260619 11:07:36.349730 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:07:36.353747 31139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.350330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.16.61.62:32969)
0619 11:07:36.350465 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.350468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.350671 (+ 203us) server_negotiation.cc:407] Connection header received
0619 11:07:36.350794 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.350796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.350888 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.350971 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:07:36.351146 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.351610 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.352491 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.352690 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.352766 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.353006 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:07:36.353370 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:07:36.353456 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.353608 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.353652 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 11:07:36.354372 30947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:07:36.357597 30947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:07:36.365656 31064 tablet_service.cc:1467] Processing CreateTablet for tablet fe1eebe73eda4ffa89f90e44ead456f0 (DEFAULT_TABLE table=test-table [id=cd7637fc568a4b01878fecc443c376f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:36.366338 31064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1eebe73eda4ffa89f90e44ead456f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:36.369143 31029 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Bootstrap starting.
I20260619 11:07:36.369326 31144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.364303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:38881 (local address 127.0.0.1:38904)
0619 11:07:36.364466 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:36.364482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:36.364560 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:36.365331 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:36.365334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:36.365352 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:36.365454 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.365460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.366867 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.366870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.367980 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.367990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.368871 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.368875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.368987 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.369034 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 11:07:36.369115 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:07:36.369211 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 11:07:36.370021 31029 tablet_bootstrap.cc:654] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:36.370046 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.364689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42509 (local address 127.0.0.1:39426)
0619 11:07:36.364926 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:36.364939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:36.365023 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:36.365741 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:36.365743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:36.365762 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:36.365849 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.365855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.367410 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.367413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.368529 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.368540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.369644 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.369647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.369727 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.369748 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:07:36.369813 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:07:36.369872 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":116,"threads_started":1}
I20260619 11:07:36.370095 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.364859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.16.61.3:42509)
0619 11:07:36.365291 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.365303 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.365323 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:07:36.365394 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.365401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.365593 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.365749 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:36.365932 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.367258 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.368677 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.369506 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.369652 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.369779 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.369843 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.369941 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":115,"threads_started":1}
I20260619 11:07:36.370556 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.364472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.16.61.2:38881)
0619 11:07:36.364813 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.364827 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.364852 (+ 25us) server_negotiation.cc:407] Connection header received
0619 11:07:36.364933 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.364941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.365174 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.365346 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:36.365579 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.366718 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.368129 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.368743 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.368853 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.368991 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.370341 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.370447 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":147,"threads_started":1}
I20260619 11:07:36.373704 31341 tablet_service.cc:1467] Processing CreateTablet for tablet fe1eebe73eda4ffa89f90e44ead456f0 (DEFAULT_TABLE table=test-table [id=cd7637fc568a4b01878fecc443c376f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:36.374888 31341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1eebe73eda4ffa89f90e44ead456f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:36.376037 31209 tablet_service.cc:1467] Processing CreateTablet for tablet fe1eebe73eda4ffa89f90e44ead456f0 (DEFAULT_TABLE table=test-table [id=cd7637fc568a4b01878fecc443c376f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:36.376502 31029 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: No bootstrap required, opened a new log
I20260619 11:07:36.376879 31029 ts_tablet_manager.cc:1397] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260619 11:07:36.377274 31209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe1eebe73eda4ffa89f90e44ead456f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:36.377650 31029 raft_consensus.cc:348] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.377801 31029 raft_consensus.cc:374] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:36.377841 31029 raft_consensus.cc:729] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcc1a7d717d4ab6adfa6ff129e82f5f, State: Initialized, Role: FOLLOWER
I20260619 11:07:36.378293 31029 consensus_queue.cc:260] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.379276 31029 ts_tablet_manager.cc:1428] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:07:36.384052 31420 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Bootstrap starting.
I20260619 11:07:36.386726 31421 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Bootstrap starting.
I20260619 11:07:36.388383 31420 tablet_bootstrap.cc:654] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:36.389408 31420 log.cc:824] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:36.389616 31421 tablet_bootstrap.cc:654] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:36.391121 31421 log.cc:824] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:36.392445 31420 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: No bootstrap required, opened a new log
I20260619 11:07:36.392684 31420 ts_tablet_manager.cc:1397] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 11:07:36.393848 31421 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: No bootstrap required, opened a new log
I20260619 11:07:36.394058 31421 ts_tablet_manager.cc:1397] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 11:07:36.397053 31420 raft_consensus.cc:348] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.397310 31420 raft_consensus.cc:374] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:36.397379 31420 raft_consensus.cc:729] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255276f3c83b4afdad2fefb4b89658e2, State: Initialized, Role: FOLLOWER
I20260619 11:07:36.397833 31420 consensus_queue.cc:260] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.398532 31407 heartbeater.cc:503] Master 127.16.61.62:32969 was elected leader, sending a full tablet report...
I20260619 11:07:36.399004 31420 ts_tablet_manager.cc:1428] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 11:07:36.398322 31421 raft_consensus.cc:348] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.399904 31421 raft_consensus.cc:374] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:36.399981 31421 raft_consensus.cc:729] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2d80f35b6d4cb2b0782cf43d7bbbf0, State: Initialized, Role: FOLLOWER
I20260619 11:07:36.400416 31421 consensus_queue.cc:260] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.401371 31421 ts_tablet_manager.cc:1428] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 11:07:36.401602 31275 heartbeater.cc:503] Master 127.16.61.62:32969 was elected leader, sending a full tablet report...
W20260619 11:07:36.436486 31138 tablet.cc:2369] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:07:36.527431 31424 raft_consensus.cc:482] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:36.527613 31424 raft_consensus.cc:504] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.528682 31424 leader_election.cc:288] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881), 4bcc1a7d717d4ab6adfa6ff129e82f5f (127.16.61.1:40579)
I20260619 11:07:36.533361 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.528973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:50429 (local address 127.16.61.2:38881)
0619 11:07:36.529091 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.529096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.529198 (+ 102us) server_negotiation.cc:407] Connection header received
0619 11:07:36.529293 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.529297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.529383 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.529469 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:36.529757 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.530554 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.532157 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.532734 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.532849 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.532921 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.533171 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.533241 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:07:36.533365 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.529043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:41073 (local address 127.16.61.1:40579)
0619 11:07:36.529288 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.529293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.529387 (+ 94us) server_negotiation.cc:407] Connection header received
0619 11:07:36.529447 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.529453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.529517 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.529631 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:36.529901 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.530640 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.532157 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.532812 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.532902 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.533038 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.533171 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.533241 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260619 11:07:36.533363 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.528864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:38881 (local address 127.16.61.3:50429)
0619 11:07:36.529072 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:36.529088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:36.529192 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:36.529498 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:36.529501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:36.529515 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:36.529629 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.529638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.530663 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.530665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.532028 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.532038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.532847 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.532849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.532980 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.533040 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 11:07:36.533123 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 11:07:36.533215 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":58}
I20260619 11:07:36.533370 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.528916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:40579 (local address 127.16.61.3:41073)
0619 11:07:36.529287 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:36.529300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:36.529383 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:36.529663 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:36.529668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:36.529689 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:36.529797 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.529801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.530743 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.530746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.532026 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:36.532038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.532910 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.532913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:36.532993 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.533069 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 11:07:36.533137 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 11:07:36.533215 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":94,"thread_start_us":119,"threads_started":1}
I20260619 11:07:36.533950 31091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1eebe73eda4ffa89f90e44ead456f0" candidate_uuid: "255276f3c83b4afdad2fefb4b89658e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" is_pre_election: true
I20260619 11:07:36.533987 31229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1eebe73eda4ffa89f90e44ead456f0" candidate_uuid: "255276f3c83b4afdad2fefb4b89658e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" is_pre_election: true
I20260619 11:07:36.534250 31229 raft_consensus.cc:2454] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 255276f3c83b4afdad2fefb4b89658e2 in term 0.
I20260619 11:07:36.534251 31091 raft_consensus.cc:2454] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 255276f3c83b4afdad2fefb4b89658e2 in term 0.
I20260619 11:07:36.534693 31296 leader_election.cc:302] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 255276f3c83b4afdad2fefb4b89658e2, cf2d80f35b6d4cb2b0782cf43d7bbbf0; no voters:
I20260619 11:07:36.534965 31424 raft_consensus.cc:2781] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:07:36.535059 31424 raft_consensus.cc:482] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:07:36.535102 31424 raft_consensus.cc:3037] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:36.536103 31424 raft_consensus.cc:504] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.536401 31424 leader_election.cc:288] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [CANDIDATE]: Term 1 election: Requested vote from peers cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881), 4bcc1a7d717d4ab6adfa6ff129e82f5f (127.16.61.1:40579)
I20260619 11:07:36.536715 31229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1eebe73eda4ffa89f90e44ead456f0" candidate_uuid: "255276f3c83b4afdad2fefb4b89658e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0"
I20260619 11:07:36.536734 31091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe1eebe73eda4ffa89f90e44ead456f0" candidate_uuid: "255276f3c83b4afdad2fefb4b89658e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f"
I20260619 11:07:36.536808 31229 raft_consensus.cc:3037] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:36.536859 31091 raft_consensus.cc:3037] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:36.537789 31229 raft_consensus.cc:2454] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 255276f3c83b4afdad2fefb4b89658e2 in term 1.
I20260619 11:07:36.537801 31091 raft_consensus.cc:2454] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 255276f3c83b4afdad2fefb4b89658e2 in term 1.
I20260619 11:07:36.538038 31296 leader_election.cc:302] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 255276f3c83b4afdad2fefb4b89658e2, 4bcc1a7d717d4ab6adfa6ff129e82f5f, cf2d80f35b6d4cb2b0782cf43d7bbbf0; no voters:
I20260619 11:07:36.538178 31424 raft_consensus.cc:2781] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:36.538422 31424 raft_consensus.cc:686] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 LEADER]: Becoming Leader. State: Replica: 255276f3c83b4afdad2fefb4b89658e2, State: Running, Role: LEADER
I20260619 11:07:36.538715 31424 consensus_queue.cc:237] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:36.541651 30948 catalog_manager.cc:5495] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3). New cstate: current_term: 1 leader_uuid: "255276f3c83b4afdad2fefb4b89658e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } health_report { overall_health: UNKNOWN } } }
I20260619 11:07:36.567384 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:36.563927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.16.61.3:42509)
0619 11:07:36.564061 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:36.564067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:36.564083 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:07:36.564277 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:36.564281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:36.564340 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:36.564444 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:07:36.564680 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.565172 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.565975 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:36.566209 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:36.566315 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:36.566614 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:07:36.566977 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:07:36.567101 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:36.567173 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 11:07:36.567224 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:07:36.572686 31091 raft_consensus.cc:1261] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Refusing update from remote peer 255276f3c83b4afdad2fefb4b89658e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:36.572691 31229 raft_consensus.cc:1261] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 1 FOLLOWER]: Refusing update from remote peer 255276f3c83b4afdad2fefb4b89658e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:36.573330 31428 consensus_queue.cc:1035] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:07:36.573473 31424 consensus_queue.cc:1035] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:07:36.576066 31408 tablet.cc:2369] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:07:36.579252 31431 mvcc.cc:205] Tried to move back new op lower bound from 7298528282915033088 to 7298528282785251328. Current Snapshot: MvccSnapshot[applied={T|T < 7298528282915033088}]
I20260619 11:07:36.579939 31433 mvcc.cc:205] Tried to move back new op lower bound from 7298528282915033088 to 7298528282785251328. Current Snapshot: MvccSnapshot[applied={T|T < 7298528282915033088}]
I20260619 11:07:36.581063 31432 mvcc.cc:205] Tried to move back new op lower bound from 7298528282915033088 to 7298528282785251328. Current Snapshot: MvccSnapshot[applied={T|T < 7298528282915033088}]
I20260619 11:07:36.587589 30948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:07:36.587766 30948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:07:36.589907 30948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Sending DeleteTablet for 3 replicas of tablet fe1eebe73eda4ffa89f90e44ead456f0
I20260619 11:07:36.590528 31209 tablet_service.cc:1514] Processing DeleteTablet for tablet fe1eebe73eda4ffa89f90e44ead456f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:36 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:38904
I20260619 11:07:36.590694 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30987
I20260619 11:07:36.590885 31341 tablet_service.cc:1514] Processing DeleteTablet for tablet fe1eebe73eda4ffa89f90e44ead456f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:36 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:39426
I20260619 11:07:36.591343 31440 tablet_replica.cc:330] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: stopping tablet replica
I20260619 11:07:36.591552 31440 raft_consensus.cc:2229] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:07:36.591816 31440 raft_consensus.cc:2258] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:36.592938 31440 ts_tablet_manager.cc:1905] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:07:36.595100 31440 ts_tablet_manager.cc:1918] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:07:36.595216 31440 log.cc:1196] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/fe1eebe73eda4ffa89f90e44ead456f0
I20260619 11:07:36.595468 31441 tablet_replica.cc:330] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: stopping tablet replica
I20260619 11:07:36.595569 31440 ts_tablet_manager.cc:1939] T fe1eebe73eda4ffa89f90e44ead456f0 P cf2d80f35b6d4cb2b0782cf43d7bbbf0: Deleting consensus metadata
I20260619 11:07:36.595892 31441 raft_consensus.cc:2229] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:07:36.596246 31441 raft_consensus.cc:2258] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:36.596751 30935 catalog_manager.cc:4841] TS cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881): tablet fe1eebe73eda4ffa89f90e44ead456f0 (table test-table [id=cd7637fc568a4b01878fecc443c376f6]) successfully deleted
I20260619 11:07:36.597594 31441 ts_tablet_manager.cc:1905] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:07:36.598917 30934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:40579 (error 108)
W20260619 11:07:36.600140 31417 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:07:36.599807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:40579 (local address 127.0.0.1:35298)
0619 11:07:36.599948 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:36.600039 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:40579: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 11:07:36.600137 31441 ts_tablet_manager.cc:1918] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:07:36.600232 31441 log.cc:1196] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/fe1eebe73eda4ffa89f90e44ead456f0
I20260619 11:07:36.600266 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 31141
I20260619 11:07:36.601001 31441 ts_tablet_manager.cc:1939] T fe1eebe73eda4ffa89f90e44ead456f0 P 255276f3c83b4afdad2fefb4b89658e2: Deleting consensus metadata
W20260619 11:07:36.600378 30934 catalog_manager.cc:4584] TS 4bcc1a7d717d4ab6adfa6ff129e82f5f (127.16.61.1:40579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe1eebe73eda4ffa89f90e44ead456f0: Network error: Client connection negotiation failed: client connection to 127.16.61.1:40579: connect: Connection reset by peer (error 104)
I20260619 11:07:36.602124 30934 catalog_manager.cc:4841] TS 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509): tablet fe1eebe73eda4ffa89f90e44ead456f0 (table test-table [id=cd7637fc568a4b01878fecc443c376f6]) successfully deleted
I20260619 11:07:36.606086 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 31277
I20260619 11:07:36.610749 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 30916
I20260619 11:07:36.617026 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:32969
--webserver_interface=127.16.61.62
--webserver_port=33905
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:36.722559 31443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:36.722793 31443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:36.722831 31443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:36.726647 31443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:36.726689 31443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:36.726708 31443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:36.726724 31443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:36.726739 31443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:36.731642 31443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:36.732591 31443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:36.733461 31443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:36.741634 31449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:36.741671 31451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:36.741843 31443 server_base.cc:1034] running on GCE node
W20260619 11:07:36.741981 31448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:36.742574 31443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:36.743769 31443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:36.744982 31443 hybrid_clock.cc:648] HybridClock initialized: now 1781867256744963 us; error 38 us; skew 500 ppm
Jun 19 11:07:36 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:36.747933 31443 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:07:36.748775 31443 webserver.cc:460] Webserver started at http://127.16.61.62:33905/ using document root <none> and password file <none>
I20260619 11:07:36.749291 31443 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:36.749351 31443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:36.753290 31443 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 11:07:36.755338 31458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:36.756637 31443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:07:36.756774 31443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "cb6e3ef97f5946b78ae8d300bf4b8927"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1f9287957b38b67d13b460082280f2ec"
server_key_iv: "8df731d56cf42c2d3f5d44cd77f24f52"
server_key_version: "encryptionkey@0"
I20260619 11:07:36.757318 31443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:36.769443 31443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:36.772454 31443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:36.772670 31443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:36.780018 31443 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:32969
I20260619 11:07:36.780355 31510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:32969 every 8 connection(s)
I20260619 11:07:36.781162 31443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:07:36.782759 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 31443
I20260619 11:07:36.783224 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:40579
--local_ip_for_outbound_sockets=127.16.61.1
--tserver_master_addrs=127.16.61.62:32969
--webserver_port=43411
--webserver_interface=127.16.61.1
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:07:36.786080 31511 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:07:36.788647 31511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Bootstrap starting.
I20260619 11:07:36.811879 31511 log.cc:824] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:36.825326 31511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:07:36.825721 31511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927: Bootstrap complete.
I20260619 11:07:36.831138 31511 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:36.831627 31511 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6e3ef97f5946b78ae8d300bf4b8927, State: Initialized, Role: FOLLOWER
I20260619 11:07:36.832211 31511 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:36.832363 31511 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:07:36.832427 31511 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:07:36.832521 31511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 2 FOLLOWER]: Advancing to term 3
I20260619 11:07:36.834874 31511 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:36.835230 31511 leader_election.cc:302] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb6e3ef97f5946b78ae8d300bf4b8927; no voters:
I20260619 11:07:36.835588 31511 leader_election.cc:288] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 11:07:36.835732 31515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 11:07:36.836011 31515 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [term 3 LEADER]: Becoming Leader. State: Replica: cb6e3ef97f5946b78ae8d300bf4b8927, State: Running, Role: LEADER
I20260619 11:07:36.836293 31515 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } }
I20260619 11:07:36.836772 31511 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:07:36.837814 31517 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb6e3ef97f5946b78ae8d300bf4b8927. Latest consensus state: current_term: 3 leader_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } } }
I20260619 11:07:36.837983 31517 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:36.837814 31516 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6e3ef97f5946b78ae8d300bf4b8927" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 32969 } } }
I20260619 11:07:36.838212 31516 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6e3ef97f5946b78ae8d300bf4b8927 [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:36.838454 31524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:07:36.843209 31524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:07:36.843918 31524 catalog_manager.cc:1259] Loaded cluster ID: 027ff8aac7844aae8782e9736991ba4c
I20260619 11:07:36.843993 31524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 11:07:36.844756 31524 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f61c9ad59ad google::LogMessage::Fail() at ??:0
@ 0x7f61c9ad888a google::LogMessage::SendToLog() at ??:0
@ 0x7f61c9ad54de google::LogMessage::Flush() at ??:0
@ 0x7f61c9ad6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f61d4414fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f61d4415d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f61d4412579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f61d4456640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f61cbd273d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f61cbd27c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f61cbd293f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f61cbd18512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f61cd35c6db start_thread at ??:0
@ 0x7f61c85d361f clone at ??:0
W20260619 11:07:36.935181 31513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:36.935413 31513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:36.935521 31513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:36.939452 31513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:36.939503 31513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:36.939589 31513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:07:36.944664 31513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:40579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=43411
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:36.945623 31513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:36.946561 31513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:36.955703 31552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:36.955729 31553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:36.955753 31555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:36.956051 31513 server_base.cc:1034] running on GCE node
I20260619 11:07:36.956429 31513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:36.957063 31513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:36.958269 31513 hybrid_clock.cc:648] HybridClock initialized: now 1781867256958237 us; error 53 us; skew 500 ppm
Jun 19 11:07:36 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:36.961288 31513 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:07:36.962371 31513 webserver.cc:460] Webserver started at http://127.16.61.1:43411/ using document root <none> and password file <none>
I20260619 11:07:36.963135 31513 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:36.963239 31513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:36.968109 31513 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:07:36.970556 31562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:36.971992 31513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:07:36.972113 31513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f"
format_stamp: "Formatted at 2026-06-19 11:07:34 on dist-test-slave-8vcz"
server_key: "1ee766a41f8cdd346234445ead274a86"
server_key_iv: "381d8105d48a7fc33ce4baac988e3506"
server_key_version: "encryptionkey@0"
I20260619 11:07:36.972774 31513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:37.002251 31513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:37.006052 31513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:37.006243 31513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:37.006934 31513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:37.008252 31569 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:07:37.012334 31513 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:07:37.012416 31513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 11:07:37.012490 31513 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:07:37.014611 31513 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:07:37.014715 31513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260619 11:07:37.014987 31569 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Bootstrap starting.
I20260619 11:07:37.025465 31513 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:40579
I20260619 11:07:37.025645 31676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:40579 every 8 connection(s)
I20260619 11:07:37.026865 31513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:07:37.031073 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 31513
I20260619 11:07:37.031701 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:38881
--local_ip_for_outbound_sockets=127.16.61.2
--tserver_master_addrs=127.16.61.62:32969
--webserver_port=41845
--webserver_interface=127.16.61.2
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:07:37.034969 31569 log.cc:824] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:37.041952 31569 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:07:37.042313 31569 tablet_bootstrap.cc:492] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Bootstrap complete.
I20260619 11:07:37.043468 31569 ts_tablet_manager.cc:1397] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.004s
I20260619 11:07:37.047784 31569 raft_consensus.cc:348] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:37.048422 31569 raft_consensus.cc:729] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcc1a7d717d4ab6adfa6ff129e82f5f, State: Initialized, Role: FOLLOWER
I20260619 11:07:37.048943 31569 consensus_queue.cc:260] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:37.050002 31569 ts_tablet_manager.cc:1428] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
W20260619 11:07:37.051316 31679 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:07:37.029373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:45435)
0619 11:07:37.029936 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.029992 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:37.031036 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:37.050441 (+ 19405us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":146,"threads_started":1}
W20260619 11:07:37.051820 31677 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:32969 (0 consecutive failures): Network error: Failed to ping master at 127.16.61.62:32969: Client connection negotiation failed: client connection to 127.16.61.62:32969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:07:37.052460 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:37.052208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:45113)
0619 11:07:37.052334 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.052400 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 11:07:37.053056 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:37.052845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:55125)
0619 11:07:37.052955 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.053001 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:07:37.053215 31677 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:07:37.142876 31680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:37.143142 31680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:37.143193 31680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:37.147151 31680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:37.147215 31680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:37.147275 31680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:07:37.152604 31680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:38881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=41845
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:37.153605 31680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:37.154556 31680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:37.163909 31690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:37.163928 31687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:37.163949 31680 server_base.cc:1034] running on GCE node
W20260619 11:07:37.163897 31688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:37.164508 31680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:37.165114 31680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:37.166311 31680 hybrid_clock.cc:648] HybridClock initialized: now 1781867257166292 us; error 28 us; skew 500 ppm
Jun 19 11:07:37 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867257, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:37.169111 31680 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:07:37.169931 31680 webserver.cc:460] Webserver started at http://127.16.61.2:41845/ using document root <none> and password file <none>
I20260619 11:07:37.170495 31680 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:37.170586 31680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:37.174273 31680 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:37.176252 31697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:37.177551 31680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:37.177666 31680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "5b937b9050d2a13d291e1122c921ab0d"
server_key_iv: "df3a4fe137527e3beebafe114acd5b32"
server_key_version: "encryptionkey@0"
I20260619 11:07:37.178220 31680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:37.194566 31680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:37.197948 31680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:37.198167 31680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:37.198764 31680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:37.199589 31680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:37.199661 31680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:37.199757 31680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:37.199802 31680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:37.208611 31680 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:38881
I20260619 11:07:37.208621 31810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:38881 every 8 connection(s)
I20260619 11:07:37.209573 31680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:07:37.213224 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:37.211606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:59437)
0619 11:07:37.212116 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.212522 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":103,"threads_started":1}
W20260619 11:07:37.213627 31811 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:32969 (0 consecutive failures): Network error: Failed to ping master at 127.16.61.62:32969: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
I20260619 11:07:37.214178 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:37.213959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:47371)
0619 11:07:37.214066 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.214129 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 11:07:37.214721 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:37.214521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:34617)
0619 11:07:37.214617 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.214662 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:07:37.214891 31811 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:07:37.219591 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 31680
I20260619 11:07:37.220033 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:42509
--local_ip_for_outbound_sockets=127.16.61.3
--tserver_master_addrs=127.16.61.62:32969
--webserver_port=46223
--webserver_interface=127.16.61.3
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:37.325866 31814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:37.326143 31814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:37.326220 31814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:37.330132 31814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:37.330212 31814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:37.330297 31814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:07:37.335270 31814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:42509
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=46223
--tserver_master_addrs=127.16.61.62:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:37.336210 31814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:37.337138 31814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:37.346179 31820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:37.346217 31822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:37.346220 31819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:37.346349 31814 server_base.cc:1034] running on GCE node
I20260619 11:07:37.346822 31814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:37.347399 31814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:37.348598 31814 hybrid_clock.cc:648] HybridClock initialized: now 1781867257348554 us; error 54 us; skew 500 ppm
Jun 19 11:07:37 dist-test-slave-8vcz krb5kdc[30368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867257, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:37.351131 31814 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:07:37.351986 31814 webserver.cc:460] Webserver started at http://127.16.61.3:46223/ using document root <none> and password file <none>
I20260619 11:07:37.352519 31814 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:37.352609 31814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:37.356256 31814 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:07:37.358227 31829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:37.359413 31814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:37.359547 31814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "255276f3c83b4afdad2fefb4b89658e2"
format_stamp: "Formatted at 2026-06-19 11:07:35 on dist-test-slave-8vcz"
server_key: "23c997fae26e8f786b3b9e8e5e9c580e"
server_key_iv: "7153b92ad65a73cb61d97943afc37503"
server_key_version: "encryptionkey@0"
I20260619 11:07:37.360110 31814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:37.375391 31814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:37.378314 31814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:37.378458 31814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:37.378914 31814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:37.379652 31814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:37.379776 31814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:37.379817 31814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:37.379837 31814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:37.388554 31814 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:42509
I20260619 11:07:37.388635 31942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:42509 every 8 connection(s)
I20260619 11:07:37.389514 31814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:07:37.393018 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:37.391497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:40883)
0619 11:07:37.391950 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.392366 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
W20260619 11:07:37.393606 31943 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:32969 (0 consecutive failures): Network error: Failed to ping master at 127.16.61.62:32969: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
I20260619 11:07:37.394282 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:37.394024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:54153)
0619 11:07:37.394164 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.394219 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 11:07:37.394938 31945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:37.394735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:43733)
0619 11:07:37.394842 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:37.394887 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:07:37.395139 31943 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:07:37.396019 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 31814
I20260619 11:07:38.054139 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:38.053711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:52561)
0619 11:07:38.053983 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:38.054035 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260619 11:07:38.215847 31948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:38.215411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:35633)
0619 11:07:38.215708 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:38.215758 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260619 11:07:38.313962 31949 raft_consensus.cc:482] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:38.314147 31949 raft_consensus.cc:504] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:38.315339 31949 leader_election.cc:288] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881), 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509)
Jun 19 11:07:38 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.2@KRBTEST.COM
Jun 19 11:07:38 dist-test-slave-8vcz krb5kdc[30368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867256, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.3@KRBTEST.COM
I20260619 11:07:38.328568 31947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:38.315533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:38881 (local address 127.16.61.1:51701)
0619 11:07:38.315735 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:38.315749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:38.315883 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:38.317542 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:38.317547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:38.318110 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:38.318576 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:38.318590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:38.319871 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:38.319874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:38.320946 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:38.320955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:38.321634 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:38.321637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:38.321861 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:38.322504 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:38.322532 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:38.324068 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:38.326327 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:38.326332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:38.326343 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:38.326632 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:38.326961 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:38.326964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:38.326966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:38.327096 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:38.327654 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:38.327660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:38.327913 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 11:07:38.328088 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 11:07:38.328398 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":145,"spinlock_wait_cycles":29568}
I20260619 11:07:38.328727 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:38.315595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42509 (local address 127.16.61.1:50465)
0619 11:07:38.316062 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:38.316079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:38.316172 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:38.317577 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:38.317581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:38.318167 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:38.318576 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:38.318591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:38.319882 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:38.319886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:38.320945 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:38.320954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:38.321630 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:38.321634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:38.321858 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:38.322532 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:38.322544 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:38.324438 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:38.326391 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:38.326395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:38.326398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:38.326663 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:38.326992 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:38.327008 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:38.327010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:38.327098 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:38.327659 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:38.327662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:38.327912 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 11:07:38.328090 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 11:07:38.328488 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":257,"spinlock_wait_cycles":10624,"thread_start_us":99,"threads_started":1}
I20260619 11:07:38.329094 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:38.315758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:51701 (local address 127.16.61.2:38881)
0619 11:07:38.316074 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:38.316099 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:38.316143 (+ 44us) server_negotiation.cc:407] Connection header received
0619 11:07:38.316860 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:38.316866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:38.317196 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:38.317490 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:38.318747 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:38.319734 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:38.321064 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:38.321507 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:38.321668 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:38.324267 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:38.324272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:38.324281 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:38.324304 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:38.326201 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:38.326744 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:38.326750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:38.326761 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:38.326860 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:38.327188 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:38.327193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:38.327195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:38.327538 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:38.327678 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:38.328601 (+ 923us) server_negotiation.cc:299] Negotiation successful
0619 11:07:38.328968 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260619 11:07:38.329278 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:38.316063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:50465 (local address 127.16.61.3:42509)
0619 11:07:38.316387 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:38.316396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:38.316425 (+ 29us) server_negotiation.cc:407] Connection header received
0619 11:07:38.317014 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:38.317019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:38.317306 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:38.318009 (+ 703us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:38.318746 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:38.319741 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:38.321062 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:38.321501 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:38.321667 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:38.324622 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:38.324626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:38.324636 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:38.324662 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:38.326276 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:38.326774 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:38.326780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:38.326792 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:38.326897 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:38.327197 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:38.327200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:38.327202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:38.327546 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:38.327701 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:38.328210 (+ 509us) server_negotiation.cc:299] Negotiation successful
0619 11:07:38.329112 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":110,"threads_started":1}
W20260619 11:07:38.331357 31564 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
W20260619 11:07:38.331521 31565 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
I20260619 11:07:38.331602 31565 leader_election.cc:302] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc1a7d717d4ab6adfa6ff129e82f5f; no voters: 255276f3c83b4afdad2fefb4b89658e2, cf2d80f35b6d4cb2b0782cf43d7bbbf0
I20260619 11:07:38.331854 31949 raft_consensus.cc:2726] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:07:38.396003 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:38.395598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:56577)
0619 11:07:38.395868 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:38.395917 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":127,"threads_started":1}
I20260619 11:07:39.055440 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:39.054833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:43891)
0619 11:07:39.055133 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:39.055198 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
I20260619 11:07:39.216838 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:39.216476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:33635)
0619 11:07:39.216712 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:39.216761 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260619 11:07:39.397203 31956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:39.396728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:43151)
0619 11:07:39.397049 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:39.397100 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260619 11:07:40.056617 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:40.056200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:39651)
0619 11:07:40.056465 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:40.056524 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260619 11:07:40.217911 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:40.217536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:52841)
0619 11:07:40.217760 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:40.217814 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260619 11:07:40.398260 31959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:40.397860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:38373)
0619 11:07:40.398112 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:40.398172 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260619 11:07:40.427416 31960 raft_consensus.cc:482] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:40.427536 31960 raft_consensus.cc:504] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:40.428015 31960 leader_election.cc:288] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881), 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509)
W20260619 11:07:40.428704 31564 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
W20260619 11:07:40.428843 31565 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
I20260619 11:07:40.428920 31565 leader_election.cc:302] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc1a7d717d4ab6adfa6ff129e82f5f; no voters: 255276f3c83b4afdad2fefb4b89658e2, cf2d80f35b6d4cb2b0782cf43d7bbbf0
I20260619 11:07:40.429086 31960 raft_consensus.cc:2726] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:07:41.057670 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:41.057294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:38121)
0619 11:07:41.057527 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:41.057579 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260619 11:07:41.218923 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:41.218564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:36109)
0619 11:07:41.218787 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:41.218839 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260619 11:07:41.399338 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:41.398986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:35449)
0619 11:07:41.399207 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:41.399255 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260619 11:07:42.058945 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:42.058536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:46351)
0619 11:07:42.058807 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:42.058857 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":69,"threads_started":1}
I20260619 11:07:42.220074 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:42.219632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:57557)
0619 11:07:42.219936 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:42.219990 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":132,"threads_started":1}
I20260619 11:07:42.400498 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:42.400088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:54113)
0619 11:07:42.400346 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:42.400401 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260619 11:07:43.060199 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:43.059810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:55611)
0619 11:07:43.060043 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:43.060106 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
I20260619 11:07:43.221256 31968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:43.220869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:38127)
0619 11:07:43.221115 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:43.221170 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260619 11:07:43.401548 31969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:43.401172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:55097)
0619 11:07:43.401403 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:43.401455 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260619 11:07:43.598407 31970 raft_consensus.cc:482] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:43.598528 31970 raft_consensus.cc:504] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:43.598963 31970 leader_election.cc:288] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881), 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509)
W20260619 11:07:43.599737 31565 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
W20260619 11:07:43.599879 31564 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
I20260619 11:07:43.599987 31564 leader_election.cc:302] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc1a7d717d4ab6adfa6ff129e82f5f; no voters: 255276f3c83b4afdad2fefb4b89658e2, cf2d80f35b6d4cb2b0782cf43d7bbbf0
I20260619 11:07:43.600169 31970 raft_consensus.cc:2726] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:07:44.061233 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:44.060845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:48999)
0619 11:07:44.061099 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:44.061148 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":104,"threads_started":1}
I20260619 11:07:44.222325 31972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:44.221908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:33341)
0619 11:07:44.222181 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:44.222240 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260619 11:07:44.402791 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:44.402259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:51455)
0619 11:07:44.402577 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:44.402674 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260619 11:07:45.062347 31974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:45.061952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:59935)
0619 11:07:45.062204 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:45.062255 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260619 11:07:45.223574 31975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:45.223123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:47823)
0619 11:07:45.223422 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:45.223479 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":77,"threads_started":1}
I20260619 11:07:45.403942 31976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:45.403501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:36291)
0619 11:07:45.403804 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:45.403856 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":80,"threads_started":1}
I20260619 11:07:45.681896 31977 raft_consensus.cc:482] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:45.682020 31977 raft_consensus.cc:504] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:45.682520 31977 leader_election.cc:288] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881), 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509)
W20260619 11:07:45.683247 31565 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
W20260619 11:07:45.683379 31564 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
I20260619 11:07:45.683465 31564 leader_election.cc:302] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc1a7d717d4ab6adfa6ff129e82f5f; no voters: 255276f3c83b4afdad2fefb4b89658e2, cf2d80f35b6d4cb2b0782cf43d7bbbf0
I20260619 11:07:45.683604 31977 raft_consensus.cc:2726] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:07:46.063565 31978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:46.063186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:49189)
0619 11:07:46.063423 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:46.063472 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260619 11:07:46.224617 31979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:46.224238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:45923)
0619 11:07:46.224475 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:46.224528 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260619 11:07:46.404987 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:46.404621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:50145)
0619 11:07:46.404849 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:46.404901 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260619 11:07:47.064714 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:47.064325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:46001)
0619 11:07:47.064557 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:47.064615 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260619 11:07:47.225684 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:47.225326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:52123)
0619 11:07:47.225547 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:47.225599 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260619 11:07:47.406142 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:47.405743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:39579)
0619 11:07:47.405994 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:47.406049 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260619 11:07:48.065740 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:48.065380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:37453)
0619 11:07:48.065597 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:48.065652 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260619 11:07:48.226737 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:48.226389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:56459)
0619 11:07:48.226600 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:48.226652 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260619 11:07:48.407215 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:48.406838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:56309)
0619 11:07:48.407072 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:48.407125 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260619 11:07:49.066780 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:49.066398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:58773)
0619 11:07:49.066636 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:49.066689 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260619 11:07:49.227957 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:49.227490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:57413)
0619 11:07:49.227804 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:49.227860 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":73,"threads_started":1}
I20260619 11:07:49.408404 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:49.408022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:36917)
0619 11:07:49.408257 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:49.408312 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260619 11:07:50.067997 31990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:50.067527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:47487)
0619 11:07:50.067846 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:50.067901 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":76,"threads_started":1}
I20260619 11:07:50.229104 31991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:50.228675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:43997)
0619 11:07:50.228946 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:50.229002 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":124,"threads_started":1}
I20260619 11:07:50.409593 31992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:50.409180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:34029)
0619 11:07:50.409441 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:50.409500 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260619 11:07:51.069064 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:51.068681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:39409)
0619 11:07:51.068921 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:51.068973 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260619 11:07:51.230216 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:51.229824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:55243)
0619 11:07:51.230071 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:51.230122 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
I20260619 11:07:51.410688 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:51.410284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.3:51035)
0619 11:07:51.410544 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:51.410598 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260619 11:07:52.070358 31996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:52.069974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.1:43437)
0619 11:07:52.070214 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:52.070267 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":82,"threads_started":1}
I20260619 11:07:52.113415 31997 raft_consensus.cc:482] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:52.113540 31997 raft_consensus.cc:504] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf2d80f35b6d4cb2b0782cf43d7bbbf0" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 38881 } } peers { permanent_uuid: "255276f3c83b4afdad2fefb4b89658e2" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42509 } } peers { permanent_uuid: "4bcc1a7d717d4ab6adfa6ff129e82f5f" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40579 } }
I20260619 11:07:52.113929 31997 leader_election.cc:288] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881), 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509)
W20260619 11:07:52.114753 31565 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf2d80f35b6d4cb2b0782cf43d7bbbf0 (127.16.61.2:38881): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
W20260619 11:07:52.114843 31564 leader_election.cc:341] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 255276f3c83b4afdad2fefb4b89658e2 (127.16.61.3:42509): Not found: Tablet not found: fe1eebe73eda4ffa89f90e44ead456f0
I20260619 11:07:52.114898 31565 leader_election.cc:302] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcc1a7d717d4ab6adfa6ff129e82f5f; no voters: 255276f3c83b4afdad2fefb4b89658e2, cf2d80f35b6d4cb2b0782cf43d7bbbf0
I20260619 11:07:52.115027 31997 raft_consensus.cc:2726] T fe1eebe73eda4ffa89f90e44ead456f0 P 4bcc1a7d717d4ab6adfa6ff129e82f5f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:07:52.231379 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:52.231003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:32969 (local address 127.16.61.2:38545)
0619 11:07:52.231240 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:52.231292 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260619 11:07:52.397564 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 31513
I20260619 11:07:52.403476 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 31680
I20260619 11:07:52.408771 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 31814
2026-06-19T11:07:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19015 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:07:52 dist-test-slave-8vcz krb5kdc[32001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:07:52 dist-test-slave-8vcz krb5kdc[32001](info): set up 2 sockets
Jun 19 11:07:52 dist-test-slave-8vcz krb5kdc[32001](info): commencing operation
krb5kdc: starting...
W20260619 11:07:53.443924 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:07:53 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:07:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:07:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:07:53.558642 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41709
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:53.663245 32014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:53.663508 32014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:53.663596 32014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:53.667459 32014 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:07:53.667536 32014 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:07:53.667567 32014 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:07:53.667598 32014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:07:53.667640 32014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:53.667698 32014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:53.667735 32014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:07:53.667768 32014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:07:53.672791 32014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:53.673703 32014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:53.674618 32014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:53.682885 32022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:53.682931 32020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:53.683133 32019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:53.683292 32014 server_base.cc:1034] running on GCE node
I20260619 11:07:53.683863 32014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:53.684864 32014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:53.686057 32014 hybrid_clock.cc:648] HybridClock initialized: now 1781867273686028 us; error 41 us; skew 500 ppm
Jun 19 11:07:53 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867273, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:53.688690 32014 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:07:53.689548 32014 webserver.cc:460] Webserver started at http://127.16.61.62:35187/ using document root <none> and password file <none>
I20260619 11:07:53.690062 32014 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:53.690153 32014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:53.690371 32014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:53.692329 32014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "d002ae89c80d412b97adeb4fd5ea8def"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "9b464f5e72e96fa57532fdd0aeaf635d"
server_key_iv: "df8ef59bb70d4dda906d7e12536a9766"
server_key_version: "encryptionkey@0"
I20260619 11:07:53.692862 32014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "d002ae89c80d412b97adeb4fd5ea8def"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "9b464f5e72e96fa57532fdd0aeaf635d"
server_key_iv: "df8ef59bb70d4dda906d7e12536a9766"
server_key_version: "encryptionkey@0"
I20260619 11:07:53.696256 32014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:07:53.698416 32029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:53.699473 32014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:07:53.699638 32014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "d002ae89c80d412b97adeb4fd5ea8def"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "9b464f5e72e96fa57532fdd0aeaf635d"
server_key_iv: "df8ef59bb70d4dda906d7e12536a9766"
server_key_version: "encryptionkey@0"
I20260619 11:07:53.699802 32014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:53.726822 32014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:53.729677 32014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:53.729864 32014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:53.737195 32014 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:41709
I20260619 11:07:53.737411 32081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:41709 every 8 connection(s)
I20260619 11:07:53.738131 32014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:07:53.740787 32082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:53.743959 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 32014
I20260619 11:07:53.744076 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:07:53.744393 16628 external_mini_cluster.cc:1428] Setting key b16c657458c3458f5f18d7fa84854977
I20260619 11:07:53.746449 32082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Bootstrap starting.
I20260619 11:07:53.748595 32082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:53.749320 32082 log.cc:824] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:53.751098 32082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: No bootstrap required, opened a new log
Jun 19 11:07:53 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:53.755416 32082 raft_consensus.cc:348] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:07:53.755681 32082 raft_consensus.cc:374] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:53.755770 32082 raft_consensus.cc:729] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d002ae89c80d412b97adeb4fd5ea8def, State: Initialized, Role: FOLLOWER
I20260619 11:07:53.756223 32082 consensus_queue.cc:260] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:07:53.756366 32082 raft_consensus.cc:388] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:07:53.756450 32082 raft_consensus.cc:482] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:07:53.756541 32082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:53.757380 32082 raft_consensus.cc:504] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:07:53.757747 32082 leader_election.cc:302] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d002ae89c80d412b97adeb4fd5ea8def; no voters:
I20260619 11:07:53.758046 32082 leader_election.cc:288] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:07:53.758152 32087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:53.758373 32087 raft_consensus.cc:686] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 1 LEADER]: Becoming Leader. State: Replica: d002ae89c80d412b97adeb4fd5ea8def, State: Running, Role: LEADER
I20260619 11:07:53.758713 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:53.745796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.16.61.62:41709)
0619 11:07:53.746218 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:53.746233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:53.746266 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:07:53.746939 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:53.746957 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:53.747340 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:53.747662 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:53.748498 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:53.749563 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:53.750419 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:53.750895 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:53.751079 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:53.753583 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:53.753589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:53.753600 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:53.753626 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:53.755496 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:53.756045 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:53.756050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:53.756054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:53.756160 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:53.756435 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:53.756438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:53.756440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:53.756912 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:53.757143 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:53.757385 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 11:07:53.757690 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":131,"threads_started":1}
I20260619 11:07:53.758742 32087 consensus_queue.cc:237] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:07:53.759373 32082 sys_catalog.cc:564] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:07:53.761847 32089 sys_catalog.cc:455] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: SysCatalogTable state changed. Reason: New leader d002ae89c80d412b97adeb4fd5ea8def. Latest consensus state: current_term: 1 leader_uuid: "d002ae89c80d412b97adeb4fd5ea8def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } } }
I20260619 11:07:53.761958 32089 sys_catalog.cc:458] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: This master's current role is: LEADER
I20260619 11:07:53.762166 32088 sys_catalog.cc:455] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d002ae89c80d412b97adeb4fd5ea8def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } } }
I20260619 11:07:53.762250 32088 sys_catalog.cc:458] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: This master's current role is: LEADER
W20260619 11:07:53.764128 32101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:07:53.764186 32101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:07:53.764263 32103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:07:53.764990 32103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:07:53.770303 32103 catalog_manager.cc:1347] Generated new cluster ID: 4b00898aff444d619ca8a3607009454d
I20260619 11:07:53.770377 32103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:07:53.787417 32103 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:07:53.788595 32103 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:07:53.801841 32103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Generated new TSK 0
I20260619 11:07:53.802496 32103 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:07:53.828691 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41709
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:53.936050 32108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:53.936280 32108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:53.936321 32108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:53.940304 32108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:53.940353 32108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:53.940438 32108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:07:53.945448 32108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:53.946408 32108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:53.947350 32108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:53.956314 32116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:53.956326 32113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:53.956317 32114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:53.956637 32108 server_base.cc:1034] running on GCE node
I20260619 11:07:53.957046 32108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:53.957629 32108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:53.958815 32108 hybrid_clock.cc:648] HybridClock initialized: now 1781867273958781 us; error 44 us; skew 500 ppm
Jun 19 11:07:53 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867273, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:53.961772 32108 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:07:53.962611 32108 webserver.cc:460] Webserver started at http://127.16.61.1:39379/ using document root <none> and password file <none>
I20260619 11:07:53.963155 32108 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:53.963243 32108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:53.963460 32108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:53.965420 32108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "e3d286a467d3430ea868fc472c405194"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "51b826b986c9776294c65f0cfe68f03a"
server_key_iv: "7d2108fa56b66eb558bc37cb26604b38"
server_key_version: "encryptionkey@0"
I20260619 11:07:53.965962 32108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "e3d286a467d3430ea868fc472c405194"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "51b826b986c9776294c65f0cfe68f03a"
server_key_iv: "7d2108fa56b66eb558bc37cb26604b38"
server_key_version: "encryptionkey@0"
I20260619 11:07:53.969453 32108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:07:53.971704 32123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:53.972813 32108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:07:53.972959 32108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "e3d286a467d3430ea868fc472c405194"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "51b826b986c9776294c65f0cfe68f03a"
server_key_iv: "7d2108fa56b66eb558bc37cb26604b38"
server_key_version: "encryptionkey@0"
I20260619 11:07:53.973069 32108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:53.991797 32108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:53.994410 32108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:53.994604 32108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:53.995155 32108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:53.995959 32108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:53.996031 32108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:53.996097 32108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:53.996151 32108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:54.005069 32108 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:32823
I20260619 11:07:54.005117 32236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:32823 every 8 connection(s)
I20260619 11:07:54.005956 32108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:07:54.014551 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 32108
I20260619 11:07:54.014727 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
Jun 19 11:07:54 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867273, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:54.015097 16628 external_mini_cluster.cc:1428] Setting key 7b920c93ace35d48beec7526d442da10
I20260619 11:07:54.019608 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.007685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:34155 (local address 127.16.61.62:41709)
0619 11:07:54.007811 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.007815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.008812 (+ 997us) server_negotiation.cc:407] Connection header received
0619 11:07:54.009665 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.009668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.009724 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.009804 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:54.010918 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.011454 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.012606 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.012842 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.012951 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.015577 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:54.015582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:54.015584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:54.015615 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:54.017214 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:54.017900 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:54.017902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:54.017904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:54.017976 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:54.018383 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:54.018385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:54.018387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:54.018645 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:54.018744 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.019322 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.019442 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:07:54.020372 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.008060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:34155)
0619 11:07:54.008668 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.008706 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:54.009464 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:54.009964 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:54.009971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:54.010309 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:54.010773 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.010787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.011583 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.011591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.012484 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.012493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.012987 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.012991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.013153 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.013744 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:54.013760 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:54.015386 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:54.017347 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:54.017356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:54.017374 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:54.017794 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:54.018070 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:54.018077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:54.018082 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:54.018282 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:54.018802 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:54.018810 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:54.019088 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 11:07:54.019281 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 11:07:54.019578 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":105,"threads_started":1}
I20260619 11:07:54.021672 32237 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41709
I20260619 11:07:54.021909 32237 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:54.022360 32237 heartbeater.cc:511] Master 127.16.61.62:41709 requested a full tablet report, sending...
I20260619 11:07:54.024113 32046 ts_manager.cc:194] Registered new tserver with Master: e3d286a467d3430ea868fc472c405194 (127.16.61.1:32823)
I20260619 11:07:54.025893 32046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:34155
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:07:54.042460 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41709
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:54.147807 32242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:54.148052 32242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:54.148099 32242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:54.152016 32242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:54.152073 32242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:54.152130 32242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:07:54.157212 32242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:54.158135 32242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:54.159044 32242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:54.168220 32250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:54.168237 32248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:54.168236 32247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:54.168375 32242 server_base.cc:1034] running on GCE node
I20260619 11:07:54.168848 32242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:54.169436 32242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:54.170595 32242 hybrid_clock.cc:648] HybridClock initialized: now 1781867274170571 us; error 40 us; skew 500 ppm
Jun 19 11:07:54 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867274, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:54.179104 32242 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:07:54.180016 32242 webserver.cc:460] Webserver started at http://127.16.61.2:36923/ using document root <none> and password file <none>
I20260619 11:07:54.180563 32242 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:54.180652 32242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:54.180874 32242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:54.182824 32242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "be16a9daefeb48cb91b0209a55ab042c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "a539a3511958a201280b4c66df216036"
server_key_iv: "dddc53a60cadf71848e93a15da5f5294"
server_key_version: "encryptionkey@0"
I20260619 11:07:54.183396 32242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "be16a9daefeb48cb91b0209a55ab042c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "a539a3511958a201280b4c66df216036"
server_key_iv: "dddc53a60cadf71848e93a15da5f5294"
server_key_version: "encryptionkey@0"
I20260619 11:07:54.186967 32242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:07:54.189373 32257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:54.190613 32242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:07:54.190755 32242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "be16a9daefeb48cb91b0209a55ab042c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "a539a3511958a201280b4c66df216036"
server_key_iv: "dddc53a60cadf71848e93a15da5f5294"
server_key_version: "encryptionkey@0"
I20260619 11:07:54.190867 32242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:54.213308 32242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:54.216207 32242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:54.216413 32242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:54.216977 32242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:54.217760 32242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:54.217831 32242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:54.217918 32242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:54.217965 32242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:54.226446 32242 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:34723
I20260619 11:07:54.226472 32370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:34723 every 8 connection(s)
I20260619 11:07:54.227272 32242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:07:54.228252 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 32242
I20260619 11:07:54.228364 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:07:54.228682 16628 external_mini_cluster.cc:1428] Setting key 8f13897b3372882b0221664cf50b4a1c
Jun 19 11:07:54 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867274, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:07:54.240983 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.229140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:58067 (local address 127.16.61.62:41709)
0619 11:07:54.229298 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.229303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.230785 (+ 1482us) server_negotiation.cc:407] Connection header received
0619 11:07:54.231637 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.231640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.231712 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.231794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:54.232937 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.233468 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.234680 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.234911 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.235023 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.237393 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:54.237403 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:54.237406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:54.237430 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:54.238859 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:54.239367 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:54.239372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:54.239374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:54.239455 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:54.239872 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:54.239875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:54.239877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:54.240058 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:54.240184 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.240688 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.240811 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 11:07:54.241937 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.229591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:58067)
0619 11:07:54.230609 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.230677 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:54.231443 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:54.231972 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:54.231978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:54.232336 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:54.232792 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.232805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.233594 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.233598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.234548 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.234558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.235058 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.235062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.235230 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.235834 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:54.235850 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:54.237231 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:54.238992 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:54.238998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:54.239012 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:54.239262 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:54.239548 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:54.239552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:54.239554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:54.239762 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:54.240232 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:54.240237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:54.240451 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 11:07:54.240641 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 11:07:54.240946 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":660,"threads_started":1}
I20260619 11:07:54.243216 32371 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41709
I20260619 11:07:54.243458 32371 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:54.243989 32371 heartbeater.cc:511] Master 127.16.61.62:41709 requested a full tablet report, sending...
I20260619 11:07:54.245100 32046 ts_manager.cc:194] Registered new tserver with Master: be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723)
I20260619 11:07:54.245960 32046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:58067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:07:54.254534 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41709
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:07:54.362210 32376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:07:54.362437 32376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:07:54.362475 32376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:07:54.366465 32376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:07:54.366516 32376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:07:54.366571 32376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:07:54.371744 32376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:07:54.373129 32376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:07:54.374557 32376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:07:54.384792 32382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:54.384801 32384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:07:54.384799 32381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:07:54.384968 32376 server_base.cc:1034] running on GCE node
I20260619 11:07:54.385406 32376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:07:54.385998 32376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:07:54.387172 32376 hybrid_clock.cc:648] HybridClock initialized: now 1781867274387134 us; error 41 us; skew 500 ppm
Jun 19 11:07:54 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867274, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:07:54.389819 32376 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:07:54.390668 32376 webserver.cc:460] Webserver started at http://127.16.61.3:41603/ using document root <none> and password file <none>
I20260619 11:07:54.391207 32376 fs_manager.cc:359] Metadata directory not provided
I20260619 11:07:54.391299 32376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:07:54.391516 32376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:07:54.393457 32376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "3360bc0f9dc8571a70bf9cf8471da5e3"
server_key_iv: "d022ed57370af37263ecae749df20b99"
server_key_version: "encryptionkey@0"
I20260619 11:07:54.394070 32376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "3360bc0f9dc8571a70bf9cf8471da5e3"
server_key_iv: "d022ed57370af37263ecae749df20b99"
server_key_version: "encryptionkey@0"
I20260619 11:07:54.397709 32376 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 11:07:54.400010 32391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:54.401172 32376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:07:54.401302 32376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "3360bc0f9dc8571a70bf9cf8471da5e3"
server_key_iv: "d022ed57370af37263ecae749df20b99"
server_key_version: "encryptionkey@0"
I20260619 11:07:54.401410 32376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:07:54.420667 32376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:07:54.423393 32376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:07:54.423585 32376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:07:54.424152 32376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:07:54.424930 32376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:07:54.425001 32376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:54.425066 32376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:07:54.425117 32376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:07:54.434105 32376 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:44987
I20260619 11:07:54.434128 32504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:44987 every 8 connection(s)
I20260619 11:07:54.434938 32376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:07:54.440403 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 32376
I20260619 11:07:54.440490 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:07:54.440817 16628 external_mini_cluster.cc:1428] Setting key 194a9625b7e27d305a95b6d26d378fc9
Jun 19 11:07:54 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867274, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:07:54.447887 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.436697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:38677 (local address 127.16.61.62:41709)
0619 11:07:54.436828 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.436833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.438032 (+ 1199us) server_negotiation.cc:407] Connection header received
0619 11:07:54.438857 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.438861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.438908 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.438987 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:54.440141 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.440744 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.442199 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.442406 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.442482 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.444976 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:54.444979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:54.444981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:54.445002 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:54.446225 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:54.446644 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:54.446646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:54.446647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:54.446714 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:54.446990 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:54.446992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:54.446993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:54.447144 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:54.447214 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.447616 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.447764 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:07:54.448618 32507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.437033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:38677)
0619 11:07:54.437885 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.437923 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:54.438638 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:54.439187 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:54.439194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:54.439528 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:07:54.440000 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.440012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.440888 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.440893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.442076 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.442085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.442554 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.442558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.442758 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.443394 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:07:54.443409 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:07:54.444836 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:07:54.446323 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:54.446327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:54.446337 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:54.446561 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:54.446791 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:07:54.446793 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:07:54.446794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:07:54.446914 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:07:54.447234 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:07:54.447239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:07:54.447433 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 11:07:54.447574 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 11:07:54.447866 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":531,"threads_started":1}
I20260619 11:07:54.449674 32505 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41709
I20260619 11:07:54.449877 32505 heartbeater.cc:464] Registering TS with master...
I20260619 11:07:54.450259 32505 heartbeater.cc:511] Master 127.16.61.62:41709 requested a full tablet report, sending...
I20260619 11:07:54.451156 32046 ts_manager.cc:194] Registered new tserver with Master: 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987)
I20260619 11:07:54.451655 32046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:38677
I20260619 11:07:54.454398 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:07:54.463366 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.456273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.16.61.62:41709)
0619 11:07:54.456409 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.456414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.456597 (+ 183us) server_negotiation.cc:407] Connection header received
0619 11:07:54.456740 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.456743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.456786 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.456865 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:07:54.457506 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.457979 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.458795 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.459004 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.459105 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.460398 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:07:54.460402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:07:54.460404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:07:54.460427 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:07:54.461738 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:54.462166 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:54.462171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:54.462172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:54.462256 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:07:54.462521 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:07:54.462526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:07:54.462528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:07:54.462746 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:07:54.462884 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.463110 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.463209 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:07:54.465996 32046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:07:54.468909 32046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:07:54.482815 32516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.476614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:32823 (local address 127.0.0.1:58730)
0619 11:07:54.477227 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.477830 (+ 603us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:54.477950 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:54.478772 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:54.478775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:54.478805 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:54.478998 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.479005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.480122 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.480125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.481382 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.481391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.482285 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.482289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.482391 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.482427 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:07:54.482538 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 11:07:54.482646 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":132,"spinlock_wait_cycles":6912,"thread_start_us":57,"threads_started":1}
I20260619 11:07:54.482815 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.476748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.16.61.1:32823)
0619 11:07:54.477853 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.477870 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.477893 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:07:54.478276 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.478282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.478466 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.478614 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:54.479147 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.479973 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.481504 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.482158 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.482268 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.482401 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.482584 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.482680 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":79,"threads_started":1}
I20260619 11:07:54.483542 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.476700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:44987 (local address 127.0.0.1:42572)
0619 11:07:54.477531 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.477588 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:54.477950 (+ 362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:54.478540 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:54.478545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:54.478566 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:54.478720 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.478734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.480132 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.480136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.481848 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.481859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.483162 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.483165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.483255 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.483278 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:07:54.483354 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:07:54.483422 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":308,"thread_start_us":96,"threads_started":1}
I20260619 11:07:54.483597 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.477086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.16.61.3:44987)
0619 11:07:54.477514 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.477529 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.477674 (+ 145us) server_negotiation.cc:407] Connection header received
0619 11:07:54.478192 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.478198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.478390 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.478547 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:54.478842 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.479975 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.481993 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.482736 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.482841 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.482980 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.483385 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.483467 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":139,"threads_started":1}
I20260619 11:07:54.483917 32521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.477429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44720 (local address 127.16.61.2:34723)
0619 11:07:54.477701 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.477711 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.478237 (+ 526us) server_negotiation.cc:407] Connection header received
0619 11:07:54.478322 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.478326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.478466 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.478613 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:54.479134 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.480227 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.481982 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.483110 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.483212 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.483337 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.483657 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.483773 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
I20260619 11:07:54.484301 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.477252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34723 (local address 127.0.0.1:44720)
0619 11:07:54.478118 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.478133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:54.478229 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:54.478772 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:54.478775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:54.478806 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:54.479002 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.479006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.480362 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.480365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.481848 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.481859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.483396 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.483400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.483485 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.483507 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:07:54.484100 (+ 593us) client_negotiation.cc:240] Negotiation successful
0619 11:07:54.484176 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":239,"spinlock_wait_cycles":6400,"thread_start_us":77,"threads_started":1}
I20260619 11:07:54.485846 32171 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab8fe3a73c546d29ee63b0e4299622d (DEFAULT_TABLE table=test-table [id=75558066b58c4f908f448ccc00e85404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:54.486137 32439 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab8fe3a73c546d29ee63b0e4299622d (DEFAULT_TABLE table=test-table [id=75558066b58c4f908f448ccc00e85404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:54.486712 32171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab8fe3a73c546d29ee63b0e4299622d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:54.487087 32439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab8fe3a73c546d29ee63b0e4299622d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:54.486797 32305 tablet_service.cc:1467] Processing CreateTablet for tablet 2ab8fe3a73c546d29ee63b0e4299622d (DEFAULT_TABLE table=test-table [id=75558066b58c4f908f448ccc00e85404]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:07:54.487694 32305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ab8fe3a73c546d29ee63b0e4299622d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:07:54.492761 32522 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Bootstrap starting.
I20260619 11:07:54.494092 32524 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Bootstrap starting.
I20260619 11:07:54.494652 32523 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Bootstrap starting.
I20260619 11:07:54.495153 32522 tablet_bootstrap.cc:654] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:54.496495 32522 log.cc:824] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:54.496748 32524 tablet_bootstrap.cc:654] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:54.496945 32523 tablet_bootstrap.cc:654] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Neither blocks nor log segments found. Creating new log.
I20260619 11:07:54.497507 32524 log.cc:824] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:54.497709 32523 log.cc:824] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:07:54.498569 32522 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: No bootstrap required, opened a new log
I20260619 11:07:54.498756 32522 ts_tablet_manager.cc:1397] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:07:54.499244 32524 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: No bootstrap required, opened a new log
I20260619 11:07:54.499410 32524 ts_tablet_manager.cc:1397] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:07:54.499405 32523 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: No bootstrap required, opened a new log
I20260619 11:07:54.499611 32523 ts_tablet_manager.cc:1397] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:07:54.502498 32523 raft_consensus.cc:348] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.502524 32522 raft_consensus.cc:348] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.502794 32523 raft_consensus.cc:374] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:54.502812 32522 raft_consensus.cc:374] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:54.502847 32523 raft_consensus.cc:729] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed8292ccf5b4c1db50fa2d55e30b57c, State: Initialized, Role: FOLLOWER
I20260619 11:07:54.502871 32522 raft_consensus.cc:729] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d286a467d3430ea868fc472c405194, State: Initialized, Role: FOLLOWER
I20260619 11:07:54.503026 32524 raft_consensus.cc:348] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.503247 32524 raft_consensus.cc:374] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:07:54.503301 32524 raft_consensus.cc:729] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be16a9daefeb48cb91b0209a55ab042c, State: Initialized, Role: FOLLOWER
I20260619 11:07:54.503293 32522 consensus_queue.cc:260] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.503293 32523 consensus_queue.cc:260] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.503787 32524 consensus_queue.cc:260] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.504084 32237 heartbeater.cc:503] Master 127.16.61.62:41709 was elected leader, sending a full tablet report...
I20260619 11:07:54.504290 32522 ts_tablet_manager.cc:1428] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:07:54.504324 32523 ts_tablet_manager.cc:1428] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:07:54.504640 32524 ts_tablet_manager.cc:1428] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:07:54.504653 32505 heartbeater.cc:503] Master 127.16.61.62:41709 was elected leader, sending a full tablet report...
I20260619 11:07:54.504956 32371 heartbeater.cc:503] Master 127.16.61.62:41709 was elected leader, sending a full tablet report...
W20260619 11:07:54.507094 32238 tablet.cc:2369] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:07:54.685891 32506 tablet.cc:2369] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:07:54.686260 32530 raft_consensus.cc:482] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:07:54.686455 32530 raft_consensus.cc:504] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.687569 32530 leader_election.cc:288] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3d286a467d3430ea868fc472c405194 (127.16.61.1:32823), 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987)
I20260619 11:07:54.691159 32518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.687843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:50901 (local address 127.16.61.1:32823)
0619 11:07:54.688002 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.688007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.688090 (+ 83us) server_negotiation.cc:407] Connection header received
0619 11:07:54.688242 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.688250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.688294 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.688374 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:54.688633 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.689220 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.690220 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.690610 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.690734 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.690798 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.690974 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.691040 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:07:54.691159 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.687787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:32823 (local address 127.16.61.2:50901)
0619 11:07:54.687976 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.687992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:54.688107 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:54.688395 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:54.688397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:54.688414 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:54.688530 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.688535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.689337 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.689343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.690085 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.690093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.690733 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.690737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.690843 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.690871 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:07:54.690960 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 11:07:54.691031 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 11:07:54.691319 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.688012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:57413 (local address 127.16.61.3:44987)
0619 11:07:54.688180 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.688184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.688269 (+ 85us) server_negotiation.cc:407] Connection header received
0619 11:07:54.688423 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.688426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.688476 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.688558 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:07:54.688817 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.689296 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.690292 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.690699 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.690825 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.690884 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.691087 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.691154 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 11:07:54.691537 32531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.687834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:44987 (local address 127.16.61.2:57413)
0619 11:07:54.688167 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.688181 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:07:54.688321 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:07:54.688568 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:07:54.688571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:07:54.688587 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:07:54.688711 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.688716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.689410 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.689415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.690169 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:07:54.690177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.690853 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.690857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:07:54.690967 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.690984 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:07:54.691081 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 11:07:54.691151 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260619 11:07:54.691831 32191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab8fe3a73c546d29ee63b0e4299622d" candidate_uuid: "be16a9daefeb48cb91b0209a55ab042c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3d286a467d3430ea868fc472c405194" is_pre_election: true
I20260619 11:07:54.692157 32191 raft_consensus.cc:2454] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be16a9daefeb48cb91b0209a55ab042c in term 0.
I20260619 11:07:54.692174 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab8fe3a73c546d29ee63b0e4299622d" candidate_uuid: "be16a9daefeb48cb91b0209a55ab042c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" is_pre_election: true
I20260619 11:07:54.692482 32459 raft_consensus.cc:2454] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be16a9daefeb48cb91b0209a55ab042c in term 0.
I20260619 11:07:54.692679 32258 leader_election.cc:302] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be16a9daefeb48cb91b0209a55ab042c, e3d286a467d3430ea868fc472c405194; no voters:
I20260619 11:07:54.692983 32530 raft_consensus.cc:2781] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:07:54.693077 32530 raft_consensus.cc:482] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:07:54.693121 32530 raft_consensus.cc:3037] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:54.694123 32530 raft_consensus.cc:504] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.694504 32530 leader_election.cc:288] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [CANDIDATE]: Term 1 election: Requested vote from peers e3d286a467d3430ea868fc472c405194 (127.16.61.1:32823), 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987)
I20260619 11:07:54.694796 32191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab8fe3a73c546d29ee63b0e4299622d" candidate_uuid: "be16a9daefeb48cb91b0209a55ab042c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3d286a467d3430ea868fc472c405194"
I20260619 11:07:54.694870 32459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ab8fe3a73c546d29ee63b0e4299622d" candidate_uuid: "be16a9daefeb48cb91b0209a55ab042c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c"
I20260619 11:07:54.694914 32191 raft_consensus.cc:3037] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:54.695029 32459 raft_consensus.cc:3037] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:07:54.696046 32191 raft_consensus.cc:2454] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be16a9daefeb48cb91b0209a55ab042c in term 1.
I20260619 11:07:54.696142 32459 raft_consensus.cc:2454] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be16a9daefeb48cb91b0209a55ab042c in term 1.
I20260619 11:07:54.696346 32258 leader_election.cc:302] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be16a9daefeb48cb91b0209a55ab042c, e3d286a467d3430ea868fc472c405194; no voters:
I20260619 11:07:54.696517 32530 raft_consensus.cc:2781] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:07:54.696697 32530 raft_consensus.cc:686] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 LEADER]: Becoming Leader. State: Replica: be16a9daefeb48cb91b0209a55ab042c, State: Running, Role: LEADER
I20260619 11:07:54.697011 32530 consensus_queue.cc:237] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:07:54.700016 32044 catalog_manager.cc:5495] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c reported cstate change: term changed from 0 to 1, leader changed from <none> to be16a9daefeb48cb91b0209a55ab042c (127.16.61.2). New cstate: current_term: 1 leader_uuid: "be16a9daefeb48cb91b0209a55ab042c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } health_report { overall_health: HEALTHY } } }
I20260619 11:07:54.717618 32521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.714392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44730 (local address 127.16.61.2:34723)
0619 11:07:54.714518 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:07:54.714522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:07:54.714533 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:07:54.714609 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:07:54.714611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:07:54.714659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:07:54.714748 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:07:54.714965 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.715439 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.716394 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:07:54.716596 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:07:54.716676 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:07:54.716925 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:07:54.717309 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:07:54.717394 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:07:54.717457 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:07:54.717502 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:07:54.723085 32191 raft_consensus.cc:1261] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Refusing update from remote peer be16a9daefeb48cb91b0209a55ab042c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:54.723080 32459 raft_consensus.cc:1261] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 1 FOLLOWER]: Refusing update from remote peer be16a9daefeb48cb91b0209a55ab042c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:07:54.723809 32530 consensus_queue.cc:1035] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:07:54.724020 32532 consensus_queue.cc:1035] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:07:54.728292 32372 tablet.cc:2369] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:07:54.730150 32535 mvcc.cc:205] Tried to move back new op lower bound from 7298528357258518528 to 7298528357161664512. Current Snapshot: MvccSnapshot[applied={T|T < 7298528357258518528}]
I20260619 11:07:54.731889 32537 mvcc.cc:205] Tried to move back new op lower bound from 7298528357258518528 to 7298528357161664512. Current Snapshot: MvccSnapshot[applied={T|T < 7298528357258518528}]
I20260619 11:07:54.737937 32046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:07:54.738128 32046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:07:54.740275 32046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Sending DeleteTablet for 3 replicas of tablet 2ab8fe3a73c546d29ee63b0e4299622d
I20260619 11:07:54.740945 32439 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab8fe3a73c546d29ee63b0e4299622d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:54 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:42572
I20260619 11:07:54.740944 32171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab8fe3a73c546d29ee63b0e4299622d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:54 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:58730
I20260619 11:07:54.741405 32544 tablet_replica.cc:330] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: stopping tablet replica
I20260619 11:07:54.741741 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 32108
I20260619 11:07:54.741736 32544 raft_consensus.cc:2229] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:07:54.741832 32305 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab8fe3a73c546d29ee63b0e4299622d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:54 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:44720
I20260619 11:07:54.742163 32544 raft_consensus.cc:2258] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:54.743712 32544 ts_tablet_manager.cc:1905] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:07:54.745810 32544 ts_tablet_manager.cc:1918] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:07:54.745921 32544 log.cc:1196] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/2ab8fe3a73c546d29ee63b0e4299622d
I20260619 11:07:54.746220 32544 ts_tablet_manager.cc:1939] T 2ab8fe3a73c546d29ee63b0e4299622d P 9ed8292ccf5b4c1db50fa2d55e30b57c: Deleting consensus metadata
I20260619 11:07:54.747184 32032 catalog_manager.cc:4841] TS 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987): tablet 2ab8fe3a73c546d29ee63b0e4299622d (table test-table [id=75558066b58c4f908f448ccc00e85404]) successfully deleted
I20260619 11:07:54.752004 32546 tablet_replica.cc:330] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: stopping tablet replica
W20260619 11:07:54.751982 32030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:32823 (error 108)
I20260619 11:07:54.752300 32546 raft_consensus.cc:2229] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:07:54.752667 32546 raft_consensus.cc:2258] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:07:54.752820 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 32242
I20260619 11:07:54.754909 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.754560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:32823 (local address 127.0.0.1:58740)
0619 11:07:54.754776 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.754844 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260619 11:07:54.755081 32030 catalog_manager.cc:4584] TS e3d286a467d3430ea868fc472c405194 (127.16.61.1:32823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ab8fe3a73c546d29ee63b0e4299622d: Network error: Client connection negotiation failed: client connection to 127.16.61.1:32823: connect: Connection refused (error 111)
I20260619 11:07:54.759778 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 32376
I20260619 11:07:54.759804 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:07:54.759517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34723 (local address 127.0.0.1:44746)
0619 11:07:54.759691 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 11:07:54.759740 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.2:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 11:07:54.765010 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 32014
I20260619 11:08:14.771044 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41709
--webserver_interface=127.16.61.62
--webserver_port=35187
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:14.878414 32548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:14.878643 32548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:14.878680 32548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:14.882586 32548 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:08:14.882628 32548 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:08:14.882643 32548 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:08:14.882658 32548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:08:14.882675 32548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:14.882699 32548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:14.882714 32548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:08:14.882745 32548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:08:14.887660 32548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=35187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:14.888572 32548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:14.889451 32548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:14.902125 32557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:14.902122 32555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:14.902388 32548 server_base.cc:1034] running on GCE node
W20260619 11:08:14.902125 32554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:14.903048 32548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:14.904222 32548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:14.905404 32548 hybrid_clock.cc:648] HybridClock initialized: now 1781867294905368 us; error 45 us; skew 500 ppm
Jun 19 11:08:14 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867294, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:14.912217 32548 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:08:14.913112 32548 webserver.cc:460] Webserver started at http://127.16.61.62:35187/ using document root <none> and password file <none>
I20260619 11:08:14.913669 32548 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:14.913769 32548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:14.917519 32548 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 11:08:14.919567 32564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:14.920756 32548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:14.920912 32548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "d002ae89c80d412b97adeb4fd5ea8def"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "9b464f5e72e96fa57532fdd0aeaf635d"
server_key_iv: "df8ef59bb70d4dda906d7e12536a9766"
server_key_version: "encryptionkey@0"
I20260619 11:08:14.921489 32548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:14.937855 32548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:14.940858 32548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:14.941043 32548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:14.947580 32548 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:41709
I20260619 11:08:14.947865 32616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:41709 every 8 connection(s)
I20260619 11:08:14.948628 32548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:08:14.952713 32617 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:08:14.954193 32617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Bootstrap starting.
I20260619 11:08:14.957702 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 32548
I20260619 11:08:14.958180 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:32823
--local_ip_for_outbound_sockets=127.16.61.1
--tserver_master_addrs=127.16.61.62:41709
--webserver_port=39379
--webserver_interface=127.16.61.1
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:08:14.971383 32617 log.cc:824] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:14.979554 32617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:08:14.979941 32617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Bootstrap complete.
I20260619 11:08:14.983946 32617 raft_consensus.cc:348] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:08:14.984310 32617 raft_consensus.cc:729] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d002ae89c80d412b97adeb4fd5ea8def, State: Initialized, Role: FOLLOWER
I20260619 11:08:14.984817 32617 consensus_queue.cc:260] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:08:14.984930 32617 raft_consensus.cc:388] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:08:14.985018 32617 raft_consensus.cc:482] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:08:14.985139 32617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 1 FOLLOWER]: Advancing to term 2
I20260619 11:08:14.986891 32617 raft_consensus.cc:504] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:08:14.987208 32617 leader_election.cc:302] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d002ae89c80d412b97adeb4fd5ea8def; no voters:
I20260619 11:08:14.987520 32617 leader_election.cc:288] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 11:08:14.987617 32621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 2 FOLLOWER]: Leader election won for term 2
I20260619 11:08:14.987854 32621 raft_consensus.cc:686] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 2 LEADER]: Becoming Leader. State: Replica: d002ae89c80d412b97adeb4fd5ea8def, State: Running, Role: LEADER
I20260619 11:08:14.988127 32621 consensus_queue.cc:237] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:08:14.988703 32617 sys_catalog.cc:564] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:08:14.988974 32621 sys_catalog.cc:455] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: SysCatalogTable state changed. Reason: New leader d002ae89c80d412b97adeb4fd5ea8def. Latest consensus state: current_term: 2 leader_uuid: "d002ae89c80d412b97adeb4fd5ea8def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } } }
I20260619 11:08:14.989077 32621 sys_catalog.cc:458] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:14.990615 32625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:08:14.991833 32622 sys_catalog.cc:455] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d002ae89c80d412b97adeb4fd5ea8def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } } }
I20260619 11:08:14.991972 32622 sys_catalog.cc:458] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:14.994668 32625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:08:14.995165 32625 catalog_manager.cc:1259] Loaded cluster ID: 4b00898aff444d619ca8a3607009454d
I20260619 11:08:14.995240 32625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:08:14.996923 32625 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:08:14.997490 32625 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Loaded TSK: 0
I20260619 11:08:15.006778 32625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Generated new TSK 1
I20260619 11:08:15.007015 32625 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:08:15.077972 32619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:15.078210 32619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:15.078260 32619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:15.082249 32619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:15.082305 32619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:15.082383 32619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:08:15.087574 32619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:32823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=39379
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:15.088573 32619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:15.089586 32619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:15.099846 32647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:15.099895 32644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:15.099846 32645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:15.100143 32619 server_base.cc:1034] running on GCE node
I20260619 11:08:15.100565 32619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:15.101307 32619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:15.102501 32619 hybrid_clock.cc:648] HybridClock initialized: now 1781867295102486 us; error 28 us; skew 500 ppm
Jun 19 11:08:15 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867295, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:15.105774 32619 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:08:15.106688 32619 webserver.cc:460] Webserver started at http://127.16.61.1:39379/ using document root <none> and password file <none>
I20260619 11:08:15.107237 32619 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:15.107343 32619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:15.111179 32619 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:15.113456 32654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:15.114745 32619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:08:15.114898 32619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "e3d286a467d3430ea868fc472c405194"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "51b826b986c9776294c65f0cfe68f03a"
server_key_iv: "7d2108fa56b66eb558bc37cb26604b38"
server_key_version: "encryptionkey@0"
I20260619 11:08:15.115419 32619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:15.136538 32619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:15.139884 32619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:15.140087 32619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:15.140667 32619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:15.141809 32661 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:08:15.145743 32619 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:08:15.145808 32619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 11:08:15.145854 32619 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:08:15.147429 32619 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:08:15.147485 32619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 11:08:15.147758 32661 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Bootstrap starting.
I20260619 11:08:15.158509 32619 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:32823
I20260619 11:08:15.158715 300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:32823 every 8 connection(s)
I20260619 11:08:15.159770 32619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:08:15.164767 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 32619
I20260619 11:08:15.165347 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:34723
--local_ip_for_outbound_sockets=127.16.61.2
--tserver_master_addrs=127.16.61.62:41709
--webserver_port=36923
--webserver_interface=127.16.61.2
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:08:15.168754 32661 log.cc:824] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:08:15 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867295, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:15.176883 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.161853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:53425 (local address 127.16.61.62:41709)
0619 11:08:15.162229 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.162249 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.162555 (+ 306us) server_negotiation.cc:407] Connection header received
0619 11:08:15.163659 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.163722 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.164039 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.164364 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:15.165600 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.166719 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.168552 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.169253 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.169448 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.171963 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:15.171967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:15.171979 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:15.172006 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:15.173719 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.174242 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.174252 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.174256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.174379 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.174725 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.174729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.174731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.175066 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:15.175205 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.175771 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.176026 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":125,"threads_started":1}
I20260619 11:08:15.176898 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.161866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:53425)
0619 11:08:15.162394 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.162434 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:15.163232 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:15.164406 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:15.164413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:15.164899 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:15.165375 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.165389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.166941 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.166946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.168412 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.168426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.169409 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.169413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.169603 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.170259 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:15.170275 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:15.171763 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:15.173849 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:15.173855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:15.173868 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:15.174122 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:15.174475 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:15.174479 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:15.174481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:15.174628 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:15.175182 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:15.175188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:15.175439 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 11:08:15.175624 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 11:08:15.175984 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":167,"threads_started":1}
I20260619 11:08:15.178413 32661 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:08:15.178851 32661 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Bootstrap complete.
I20260619 11:08:15.179350 301 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41709
I20260619 11:08:15.179579 301 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:15.180143 301 heartbeater.cc:511] Master 127.16.61.62:41709 requested a full tablet report, sending...
I20260619 11:08:15.180251 32661 ts_tablet_manager.cc:1397] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.018s
I20260619 11:08:15.182747 32581 ts_manager.cc:194] Registered new tserver with Master: e3d286a467d3430ea868fc472c405194 (127.16.61.1:32823)
I20260619 11:08:15.183125 32661 raft_consensus.cc:348] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.183626 32661 raft_consensus.cc:729] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d286a467d3430ea868fc472c405194, State: Initialized, Role: FOLLOWER
I20260619 11:08:15.184128 32661 consensus_queue.cc:260] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.185192 32661 ts_tablet_manager.cc:1428] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 11:08:15.186808 32581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:53425
I20260619 11:08:15.189461 301 heartbeater.cc:503] Master 127.16.61.62:41709 was elected leader, sending a full tablet report...
Jun 19 11:08:15 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867294, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.1@KRBTEST.COM
I20260619 11:08:15.197160 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.185692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.16.61.1:32823)
0619 11:08:15.187584 (+ 1892us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.187593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.187612 (+ 19us) server_negotiation.cc:407] Connection header received
0619 11:08:15.187716 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.187721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.187929 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.188061 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:15.188794 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.189896 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.190861 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.191364 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.191481 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.193666 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:15.193675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:15.193687 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:15.193711 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:15.195272 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.195797 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.195803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.195805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.195888 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.196165 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.196170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.196171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.196400 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:15.196540 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.196868 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.197019 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1801,"thread_start_us":85,"threads_started":1}
I20260619 11:08:15.197158 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.185581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:32823 (local address 127.0.0.1:48590)
0619 11:08:15.186001 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.186051 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:15.186172 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:15.188087 (+ 1915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:15.188093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:15.188535 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:15.188667 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.188677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.190104 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.190106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.190717 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.190725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.191483 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.191486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.191607 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.192224 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:15.192237 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:15.193535 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:15.195413 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:15.195421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:15.195426 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:15.195648 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:15.195991 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:15.195994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:15.195996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:15.196078 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:15.196520 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:15.196527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:15.196648 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 11:08:15.196778 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 11:08:15.196974 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
I20260619 11:08:15.198068 32703 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab8fe3a73c546d29ee63b0e4299622d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:54 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:48590
I20260619 11:08:15.198858 310 tablet_replica.cc:330] stopping tablet replica
I20260619 11:08:15.199136 310 raft_consensus.cc:2229] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:15.199368 310 raft_consensus.cc:2258] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:15.200572 310 ts_tablet_manager.cc:1905] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:15.204721 310 ts_tablet_manager.cc:1918] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:15.204818 310 log.cc:1196] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal/wals/2ab8fe3a73c546d29ee63b0e4299622d
I20260619 11:08:15.205093 310 ts_tablet_manager.cc:1939] T 2ab8fe3a73c546d29ee63b0e4299622d P e3d286a467d3430ea868fc472c405194: Deleting consensus metadata
I20260619 11:08:15.206887 32565 catalog_manager.cc:4841] TS e3d286a467d3430ea868fc472c405194 (127.16.61.1:32823): tablet 2ab8fe3a73c546d29ee63b0e4299622d (table test-table [id=75558066b58c4f908f448ccc00e85404]) successfully deleted
W20260619 11:08:15.282459 305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:15.282738 305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:15.282814 305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:15.286736 305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:15.286820 305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:15.286921 305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:08:15.292258 305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:34723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=36923
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:15.293396 305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:15.294412 305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:15.304035 318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:15.304088 316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:15.304106 315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:15.304386 305 server_base.cc:1034] running on GCE node
I20260619 11:08:15.304790 305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:15.305401 305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:15.306589 305 hybrid_clock.cc:648] HybridClock initialized: now 1781867295306554 us; error 37 us; skew 500 ppm
Jun 19 11:08:15 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867295, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:15.309208 305 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:08:15.310062 305 webserver.cc:460] Webserver started at http://127.16.61.2:36923/ using document root <none> and password file <none>
I20260619 11:08:15.310607 305 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:15.310730 305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:15.314666 305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:08:15.316753 325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:15.317922 305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:08:15.318064 305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "be16a9daefeb48cb91b0209a55ab042c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "a539a3511958a201280b4c66df216036"
server_key_iv: "dddc53a60cadf71848e93a15da5f5294"
server_key_version: "encryptionkey@0"
I20260619 11:08:15.318619 305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:15.334820 305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:15.337661 305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:15.337857 305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:15.338402 305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:15.339409 332 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:08:15.343405 305 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:08:15.343497 305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 11:08:15.343581 305 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:08:15.345206 305 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:08:15.345274 305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 11:08:15.345569 332 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Bootstrap starting.
I20260619 11:08:15.355511 305 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:34723
I20260619 11:08:15.355556 439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:34723 every 8 connection(s)
I20260619 11:08:15.356557 305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:08:15.362663 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 305
I20260619 11:08:15.363130 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:44987
--local_ip_for_outbound_sockets=127.16.61.3
--tserver_master_addrs=127.16.61.62:41709
--webserver_port=41603
--webserver_interface=127.16.61.3
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:08:15.364464 332 log.cc:824] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:08:15 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867295, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:15.371762 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.358351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:44409 (local address 127.16.61.62:41709)
0619 11:08:15.358521 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.358526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.359193 (+ 667us) server_negotiation.cc:407] Connection header received
0619 11:08:15.360391 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.360393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.360438 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.360512 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:15.362097 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.362644 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.364053 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.364288 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.364409 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.367356 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:15.367363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:15.367369 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:15.367405 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:15.369291 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.369895 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.369901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.369904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.369998 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.370399 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.370404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.370405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.370664 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:15.370794 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.371350 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.371500 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:08:15.372498 332 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:08:15.372540 442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.358431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:44409)
0619 11:08:15.358981 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.359040 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:15.360131 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:15.360708 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:15.360724 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:15.361222 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:15.361917 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.361940 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.362869 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.362873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.363926 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.363936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.364421 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.364425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.364657 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.365278 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:15.365304 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:15.367159 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:15.369422 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:15.369427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:15.369439 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:15.369783 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:15.370110 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:15.370113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:15.370115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:15.370287 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:15.370799 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:15.370804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:15.371073 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 11:08:15.371273 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 11:08:15.371566 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260619 11:08:15.372938 332 tablet_bootstrap.cc:492] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Bootstrap complete.
I20260619 11:08:15.373679 440 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41709
I20260619 11:08:15.373922 440 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:15.374114 332 ts_tablet_manager.cc:1397] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.004s
I20260619 11:08:15.374528 440 heartbeater.cc:511] Master 127.16.61.62:41709 requested a full tablet report, sending...
I20260619 11:08:15.376406 32581 ts_manager.cc:194] Registered new tserver with Master: be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723)
I20260619 11:08:15.376267 332 raft_consensus.cc:348] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.376741 332 raft_consensus.cc:729] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: be16a9daefeb48cb91b0209a55ab042c, State: Initialized, Role: FOLLOWER
I20260619 11:08:15.377233 332 consensus_queue.cc:260] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.377616 32581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:44409
I20260619 11:08:15.378211 332 ts_tablet_manager.cc:1428] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260619 11:08:15.380038 440 heartbeater.cc:503] Master 127.16.61.62:41709 was elected leader, sending a full tablet report...
I20260619 11:08:15.383991 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.378007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34723 (local address 127.0.0.1:39286)
0619 11:08:15.378107 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.378123 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:15.378190 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:15.380299 (+ 2109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:15.380302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:15.380319 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:15.380413 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.380421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.381431 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.381437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.382770 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.382784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.383514 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.383517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.383622 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.383713 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 11:08:15.383786 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 11:08:15.383864 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 11:08:15.384028 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.378119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.16.61.2:34723)
0619 11:08:15.379884 (+ 1765us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.379892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.379910 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:08:15.379960 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.379964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.380125 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.380301 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:15.380552 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.381324 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.382893 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.383396 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.383484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.383574 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.383862 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.383937 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689,"thread_start_us":111,"threads_started":1}
I20260619 11:08:15.384900 374 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ab8fe3a73c546d29ee63b0e4299622d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:07:54 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:39286
I20260619 11:08:15.385646 447 tablet_replica.cc:330] stopping tablet replica
I20260619 11:08:15.385934 447 raft_consensus.cc:2229] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:15.386163 447 raft_consensus.cc:2258] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:15.387144 447 ts_tablet_manager.cc:1905] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:15.391506 447 ts_tablet_manager.cc:1918] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:15.391597 447 log.cc:1196] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/2ab8fe3a73c546d29ee63b0e4299622d
I20260619 11:08:15.391873 447 ts_tablet_manager.cc:1939] T 2ab8fe3a73c546d29ee63b0e4299622d P be16a9daefeb48cb91b0209a55ab042c: Deleting consensus metadata
I20260619 11:08:15.393414 32565 catalog_manager.cc:4841] TS be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723): tablet 2ab8fe3a73c546d29ee63b0e4299622d (table test-table [id=75558066b58c4f908f448ccc00e85404]) successfully deleted
W20260619 11:08:15.478456 443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:15.478693 443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:15.478731 443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:15.482710 443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:15.482762 443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:15.482817 443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:08:15.487843 443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:44987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=41603
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:15.488736 443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:15.489795 443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:15.499109 453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:15.499123 452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:15.499251 443 server_base.cc:1034] running on GCE node
W20260619 11:08:15.499338 455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:15.499797 443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:15.500403 443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:15.501595 443 hybrid_clock.cc:648] HybridClock initialized: now 1781867295501578 us; error 26 us; skew 500 ppm
Jun 19 11:08:15 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867295, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:15.504340 443 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:08:15.505191 443 webserver.cc:460] Webserver started at http://127.16.61.3:41603/ using document root <none> and password file <none>
I20260619 11:08:15.505718 443 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:15.505817 443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:15.509543 443 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:15.511464 462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:15.512672 443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:15.512799 443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "3360bc0f9dc8571a70bf9cf8471da5e3"
server_key_iv: "d022ed57370af37263ecae749df20b99"
server_key_version: "encryptionkey@0"
I20260619 11:08:15.513315 443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:15.538273 443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:15.541009 443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:15.541203 443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:15.541744 443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:15.542533 443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:15.542603 443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:15.542670 443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:15.542719 443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:15.551635 443 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:44987
I20260619 11:08:15.551693 575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:44987 every 8 connection(s)
I20260619 11:08:15.552619 443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:08:15.560173 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 443
Jun 19 11:08:15 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867295, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:15.565994 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.554357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:47865 (local address 127.16.61.62:41709)
0619 11:08:15.554536 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.554542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.555357 (+ 815us) server_negotiation.cc:407] Connection header received
0619 11:08:15.556733 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.556736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.556783 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.556866 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:15.557941 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.558475 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.559532 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.559787 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.559880 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.562479 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:15.562489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:15.562492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:15.562514 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:15.564019 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.564521 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.564525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.564527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.564606 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.564946 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.564949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.564951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.565119 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:15.565224 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.565709 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.565817 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:08:15.566815 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.554720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:47865)
0619 11:08:15.555177 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.555227 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:15.556531 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:15.557031 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:15.557038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:15.557361 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:15.557800 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.557812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.558578 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.558580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.559417 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.559426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.559930 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.559933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.560067 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.560683 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:15.560699 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:15.562310 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:15.564137 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:15.564143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:15.564158 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:15.564415 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:15.564698 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:15.564702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:15.564704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:15.564845 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:15.565223 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:15.565230 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:15.565472 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 11:08:15.565657 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 11:08:15.565957 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":143,"threads_started":1}
I20260619 11:08:15.568061 576 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41709
I20260619 11:08:15.568323 576 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:15.568719 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.560834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.16.61.62:41709)
0619 11:08:15.561056 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.561060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.561186 (+ 126us) server_negotiation.cc:407] Connection header received
0619 11:08:15.561272 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.561277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.561334 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.561449 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:15.562245 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.562775 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.563622 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.563929 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.564040 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.565356 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:15.565360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:15.565362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:15.565386 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:15.566904 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.567418 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.567421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.567422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.567509 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.567823 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.567826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.567828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.568032 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:15.568243 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.568444 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.568547 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260619 11:08:15.568892 576 heartbeater.cc:511] Master 127.16.61.62:41709 requested a full tablet report, sending...
I20260619 11:08:15.569965 32581 ts_manager.cc:194] Registered new tserver with Master: 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987)
I20260619 11:08:15.570645 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:08:15.570631 32581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:47865
I20260619 11:08:15.574286 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.571235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.16.61.62:41709)
0619 11:08:15.571339 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.571342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.571499 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:08:15.571646 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.571649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.571786 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.571864 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:08:15.572110 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.572559 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.573430 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.573630 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.573700 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.573938 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:08:15.574042 (+ 104us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 11:08:15.574210 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37832: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 11:08:15.574358 580 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37832: token expired
W20260619 11:08:15.574721 32509 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.61.62:41709)
I20260619 11:08:15.581323 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.575067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.16.61.62:41709)
0619 11:08:15.575169 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.575172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.575209 (+ 37us) server_negotiation.cc:407] Connection header received
0619 11:08:15.575286 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.575287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.575316 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.575370 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:15.575982 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.576430 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.577172 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.577377 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.577461 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.578690 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:15.578693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:15.578694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:15.578710 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:15.580063 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.580438 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.580440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.580441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.580507 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:15.580733 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:15.580735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:15.580736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:15.580875 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:15.580971 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.581150 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.581229 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:08:15.583379 32581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:08:15.587098 32581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:08:15.594463 32703 tablet_service.cc:1467] Processing CreateTablet for tablet 157ef5b6244f484193a7ba4c505fa3e6 (DEFAULT_TABLE table=test-table [id=eefd05c82e7447abb9d1272004d95062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:15.594440 374 tablet_service.cc:1467] Processing CreateTablet for tablet 157ef5b6244f484193a7ba4c505fa3e6 (DEFAULT_TABLE table=test-table [id=eefd05c82e7447abb9d1272004d95062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:15.595027 32703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157ef5b6244f484193a7ba4c505fa3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:15.595059 374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157ef5b6244f484193a7ba4c505fa3e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:15.597482 332 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: Bootstrap starting.
I20260619 11:08:15.598011 32661 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Bootstrap starting.
I20260619 11:08:15.598367 332 tablet_bootstrap.cc:654] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:15.598862 32661 tablet_bootstrap.cc:654] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:15.599331 332 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: No bootstrap required, opened a new log
I20260619 11:08:15.599433 332 ts_tablet_manager.cc:1397] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 11:08:15.599908 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.593514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:44987 (local address 127.0.0.1:57102)
0619 11:08:15.593669 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.593683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:15.593754 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:15.594693 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:15.594695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:15.594712 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:15.594819 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.594825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.596360 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.596367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.597485 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.597498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.599541 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.599545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.599638 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.599699 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 11:08:15.599764 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 11:08:15.599821 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 11:08:15.600168 332 raft_consensus.cc:348] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.600291 332 raft_consensus.cc:374] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:15.600338 332 raft_consensus.cc:729] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be16a9daefeb48cb91b0209a55ab042c, State: Initialized, Role: FOLLOWER
I20260619 11:08:15.600453 332 consensus_queue.cc:260] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.600515 32661 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: No bootstrap required, opened a new log
I20260619 11:08:15.600589 32661 ts_tablet_manager.cc:1397] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 11:08:15.600719 332 ts_tablet_manager.cc:1428] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260619 11:08:15.601078 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.593721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.16.61.3:44987)
0619 11:08:15.594090 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.594101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.594125 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:08:15.594204 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.594209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.594523 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.594970 (+ 447us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:15.595150 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.596228 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.598479 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.599392 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.600208 (+ 816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.600367 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.600420 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.600510 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":161,"threads_started":1}
I20260619 11:08:15.601012 32661 raft_consensus.cc:348] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.601128 32661 raft_consensus.cc:374] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:15.601171 32661 raft_consensus.cc:729] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d286a467d3430ea868fc472c405194, State: Initialized, Role: FOLLOWER
I20260619 11:08:15.601315 32661 consensus_queue.cc:260] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.601572 32661 ts_tablet_manager.cc:1428] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 11:08:15.604233 510 tablet_service.cc:1467] Processing CreateTablet for tablet 157ef5b6244f484193a7ba4c505fa3e6 (DEFAULT_TABLE table=test-table [id=eefd05c82e7447abb9d1272004d95062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:15.605152 510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 157ef5b6244f484193a7ba4c505fa3e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260619 11:08:15.607545 441 tablet.cc:2369] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:08:15.611263 588 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Bootstrap starting.
I20260619 11:08:15.613372 588 tablet_bootstrap.cc:654] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:15.614403 588 log.cc:824] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:15.616062 588 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: No bootstrap required, opened a new log
I20260619 11:08:15.616252 588 ts_tablet_manager.cc:1397] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:08:15.619390 588 raft_consensus.cc:348] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.619616 588 raft_consensus.cc:374] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:15.619714 588 raft_consensus.cc:729] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed8292ccf5b4c1db50fa2d55e30b57c, State: Initialized, Role: FOLLOWER
I20260619 11:08:15.620146 588 consensus_queue.cc:260] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.620779 576 heartbeater.cc:503] Master 127.16.61.62:41709 was elected leader, sending a full tablet report...
I20260619 11:08:15.620970 588 ts_tablet_manager.cc:1428] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 11:08:15.661003 302 tablet.cc:2369] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:08:15.803576 577 tablet.cc:2369] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:08:15.810384 307 raft_consensus.cc:482] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:15.810575 307 raft_consensus.cc:504] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.811575 307 leader_election.cc:288] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987), be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723)
I20260619 11:08:15.815408 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.811974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:55665 (local address 127.16.61.2:34723)
0619 11:08:15.812117 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.812121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.812240 (+ 119us) server_negotiation.cc:407] Connection header received
0619 11:08:15.812449 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.812453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.812519 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.812604 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:15.812868 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.813377 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.814431 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.814821 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.814937 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.814994 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.815218 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.815284 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:08:15.815418 591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.811857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34723 (local address 127.16.61.1:55665)
0619 11:08:15.812132 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.812148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:15.812313 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:15.812618 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:15.812621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:15.812639 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:15.812752 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.812757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.813487 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.813492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.814304 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.814312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.814942 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.814945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.815033 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.815113 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 11:08:15.815203 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:08:15.815276 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":21,"thread_start_us":91,"threads_started":1}
I20260619 11:08:15.815496 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.811985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:40399 (local address 127.16.61.3:44987)
0619 11:08:15.812125 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.812128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.812495 (+ 367us) server_negotiation.cc:407] Connection header received
0619 11:08:15.812656 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.812659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.812707 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.812792 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:15.813035 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.813515 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.814501 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.814901 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.815014 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.815079 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.815303 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.815368 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:08:15.815557 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.811857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:44987 (local address 127.16.61.1:40399)
0619 11:08:15.812382 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.812394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:15.812537 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:15.812803 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:15.812806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:15.812821 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:15.812933 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.812938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.813625 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.813628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.814380 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:15.814388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.815030 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.815034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:15.815178 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.815195 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:08:15.815288 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 11:08:15.815354 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":299,"thread_start_us":88,"threads_started":1}
I20260619 11:08:15.816181 393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157ef5b6244f484193a7ba4c505fa3e6" candidate_uuid: "e3d286a467d3430ea868fc472c405194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be16a9daefeb48cb91b0209a55ab042c" is_pre_election: true
I20260619 11:08:15.816183 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157ef5b6244f484193a7ba4c505fa3e6" candidate_uuid: "e3d286a467d3430ea868fc472c405194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" is_pre_election: true
I20260619 11:08:15.816500 393 raft_consensus.cc:2454] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3d286a467d3430ea868fc472c405194 in term 0.
I20260619 11:08:15.816514 530 raft_consensus.cc:2454] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3d286a467d3430ea868fc472c405194 in term 0.
I20260619 11:08:15.816942 32655 leader_election.cc:302] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be16a9daefeb48cb91b0209a55ab042c, e3d286a467d3430ea868fc472c405194; no voters:
I20260619 11:08:15.817209 307 raft_consensus.cc:2781] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:08:15.817296 307 raft_consensus.cc:482] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:08:15.817339 307 raft_consensus.cc:3037] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:15.818406 307 raft_consensus.cc:504] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.818768 307 leader_election.cc:288] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 1 election: Requested vote from peers 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987), be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723)
I20260619 11:08:15.819072 530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157ef5b6244f484193a7ba4c505fa3e6" candidate_uuid: "e3d286a467d3430ea868fc472c405194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c"
I20260619 11:08:15.819092 393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "157ef5b6244f484193a7ba4c505fa3e6" candidate_uuid: "e3d286a467d3430ea868fc472c405194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be16a9daefeb48cb91b0209a55ab042c"
I20260619 11:08:15.819198 530 raft_consensus.cc:3037] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:15.819227 393 raft_consensus.cc:3037] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:15.820290 393 raft_consensus.cc:2454] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d286a467d3430ea868fc472c405194 in term 1.
I20260619 11:08:15.820433 530 raft_consensus.cc:2454] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3d286a467d3430ea868fc472c405194 in term 1.
I20260619 11:08:15.820549 32655 leader_election.cc:302] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be16a9daefeb48cb91b0209a55ab042c, e3d286a467d3430ea868fc472c405194; no voters:
I20260619 11:08:15.820715 307 raft_consensus.cc:2781] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:15.820919 307 raft_consensus.cc:686] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 LEADER]: Becoming Leader. State: Replica: e3d286a467d3430ea868fc472c405194, State: Running, Role: LEADER
I20260619 11:08:15.821167 307 consensus_queue.cc:237] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:15.824223 32580 catalog_manager.cc:5495] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3d286a467d3430ea868fc472c405194 (127.16.61.1). New cstate: current_term: 1 leader_uuid: "e3d286a467d3430ea868fc472c405194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } health_report { overall_health: UNKNOWN } } }
I20260619 11:08:15.834132 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.830939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.16.61.1:32823)
0619 11:08:15.831187 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:15.831193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:15.831204 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:08:15.831255 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:15.831257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:15.831299 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:15.831393 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:08:15.831583 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.832104 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.832885 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:15.833115 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:15.833188 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:15.833410 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:08:15.833768 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:08:15.833856 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:15.833919 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:08:15.833957 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260619 11:08:15.837539 393 raft_consensus.cc:1261] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Refusing update from remote peer e3d286a467d3430ea868fc472c405194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:15.837544 530 raft_consensus.cc:1261] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 1 FOLLOWER]: Refusing update from remote peer e3d286a467d3430ea868fc472c405194: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:15.838173 593 consensus_queue.cc:1035] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:15.838375 307 consensus_queue.cc:1035] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:15.845052 598 mvcc.cc:205] Tried to move back new op lower bound from 7298528443745452032 to 7298528443686064128. Current Snapshot: MvccSnapshot[applied={T|T < 7298528443745452032}]
I20260619 11:08:15.846601 599 mvcc.cc:205] Tried to move back new op lower bound from 7298528443745452032 to 7298528443686064128. Current Snapshot: MvccSnapshot[applied={T|T < 7298528443745452032}]
I20260619 11:08:15.851581 32581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:15.851778 32581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:15.853910 32581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Sending DeleteTablet for 3 replicas of tablet 157ef5b6244f484193a7ba4c505fa3e6
I20260619 11:08:15.854526 510 tablet_service.cc:1514] Processing DeleteTablet for tablet 157ef5b6244f484193a7ba4c505fa3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:15 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:57102
I20260619 11:08:15.854679 32703 tablet_service.cc:1514] Processing DeleteTablet for tablet 157ef5b6244f484193a7ba4c505fa3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:15 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:48590
I20260619 11:08:15.855017 607 tablet_replica.cc:330] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: stopping tablet replica
I20260619 11:08:15.855080 374 tablet_service.cc:1514] Processing DeleteTablet for tablet 157ef5b6244f484193a7ba4c505fa3e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:15 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:39286
I20260619 11:08:15.855175 607 raft_consensus.cc:2229] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:08:15.855248 447 tablet_replica.cc:330] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: stopping tablet replica
I20260619 11:08:15.855306 606 tablet_replica.cc:330] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: stopping tablet replica
I20260619 11:08:15.855357 447 raft_consensus.cc:2229] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:15.855436 607 raft_consensus.cc:2258] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:15.855464 447 raft_consensus.cc:2258] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:15.855566 606 raft_consensus.cc:2229] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:15.856063 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 32619
I20260619 11:08:15.856138 606 raft_consensus.cc:2258] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:15.856273 607 ts_tablet_manager.cc:1905] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:15.857599 606 ts_tablet_manager.cc:1905] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:15.860261 447 ts_tablet_manager.cc:1905] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:15.860968 606 ts_tablet_manager.cc:1918] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:15.861064 606 log.cc:1196] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/157ef5b6244f484193a7ba4c505fa3e6
I20260619 11:08:15.861420 606 ts_tablet_manager.cc:1939] T 157ef5b6244f484193a7ba4c505fa3e6 P 9ed8292ccf5b4c1db50fa2d55e30b57c: Deleting consensus metadata
I20260619 11:08:15.862190 447 ts_tablet_manager.cc:1918] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:15.862290 447 log.cc:1196] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/157ef5b6244f484193a7ba4c505fa3e6
I20260619 11:08:15.862577 447 ts_tablet_manager.cc:1939] T 157ef5b6244f484193a7ba4c505fa3e6 P be16a9daefeb48cb91b0209a55ab042c: Deleting consensus metadata
I20260619 11:08:15.862574 32567 catalog_manager.cc:4841] TS 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987): tablet 157ef5b6244f484193a7ba4c505fa3e6 (table test-table [id=eefd05c82e7447abb9d1272004d95062]) successfully deleted
I20260619 11:08:15.863165 32565 catalog_manager.cc:4841] TS be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723): tablet 157ef5b6244f484193a7ba4c505fa3e6 (table test-table [id=eefd05c82e7447abb9d1272004d95062]) successfully deleted
W20260619 11:08:15.864636 32565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:32823 (error 108)
I20260619 11:08:15.865360 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 305
I20260619 11:08:15.865674 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:15.865355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:32823 (local address 127.0.0.1:48616)
0619 11:08:15.865503 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:15.865582 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:32823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:08:15.866003 32565 catalog_manager.cc:4584] TS e3d286a467d3430ea868fc472c405194 (127.16.61.1:32823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 157ef5b6244f484193a7ba4c505fa3e6: Network error: Client connection negotiation failed: client connection to 127.16.61.1:32823: connect: Connection refused (error 111)
I20260619 11:08:15.871521 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 443
I20260619 11:08:15.876488 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 32548
I20260619 11:08:35.882653 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41709
--webserver_interface=127.16.61.62
--webserver_port=35187
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:35.986387 609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:35.986618 609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:35.986655 609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:35.990437 609 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 11:08:35.990475 609 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 11:08:35.990490 609 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 11:08:35.990504 609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:08:35.990521 609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:35.990538 609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:35.990553 609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:08:35.990568 609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:08:35.995417 609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:41709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=35187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:35.996304 609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:35.997165 609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:36.005241 615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:36.005326 609 server_base.cc:1034] running on GCE node
W20260619 11:08:36.005338 614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:36.005265 617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:36.006117 609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:36.007138 609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:36.008291 609 hybrid_clock.cc:648] HybridClock initialized: now 1781867316008278 us; error 26 us; skew 500 ppm
Jun 19 11:08:36 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:36.015763 609 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:08:36.016605 609 webserver.cc:460] Webserver started at http://127.16.61.62:35187/ using document root <none> and password file <none>
I20260619 11:08:36.017134 609 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:36.017194 609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:36.020862 609 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:36.022910 624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:36.024096 609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:36.024227 609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "d002ae89c80d412b97adeb4fd5ea8def"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "9b464f5e72e96fa57532fdd0aeaf635d"
server_key_iv: "df8ef59bb70d4dda906d7e12536a9766"
server_key_version: "encryptionkey@0"
I20260619 11:08:36.024731 609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:36.040791 609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:36.043382 609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:36.043568 609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:36.050269 609 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:41709
I20260619 11:08:36.050542 676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:41709 every 8 connection(s)
I20260619 11:08:36.051270 609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:08:36.055399 677 sys_catalog.cc:263] Verifying existing consensus state
I20260619 11:08:36.057058 677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Bootstrap starting.
I20260619 11:08:36.058357 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 609
I20260619 11:08:36.058841 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:32823
--local_ip_for_outbound_sockets=127.16.61.1
--tserver_master_addrs=127.16.61.62:41709
--webserver_port=39379
--webserver_interface=127.16.61.1
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:08:36.075623 677 log.cc:824] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:36.086751 677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:08:36.087126 677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Bootstrap complete.
I20260619 11:08:36.091001 677 raft_consensus.cc:348] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:08:36.091372 677 raft_consensus.cc:729] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d002ae89c80d412b97adeb4fd5ea8def, State: Initialized, Role: FOLLOWER
I20260619 11:08:36.091881 677 consensus_queue.cc:260] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:08:36.091995 677 raft_consensus.cc:388] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:08:36.092075 677 raft_consensus.cc:482] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:08:36.092178 677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 2 FOLLOWER]: Advancing to term 3
I20260619 11:08:36.094029 677 raft_consensus.cc:504] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:08:36.094342 677 leader_election.cc:302] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d002ae89c80d412b97adeb4fd5ea8def; no voters:
I20260619 11:08:36.094648 677 leader_election.cc:288] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 11:08:36.094725 681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 3 FOLLOWER]: Leader election won for term 3
I20260619 11:08:36.094955 681 raft_consensus.cc:686] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [term 3 LEADER]: Becoming Leader. State: Replica: d002ae89c80d412b97adeb4fd5ea8def, State: Running, Role: LEADER
I20260619 11:08:36.095285 681 consensus_queue.cc:237] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } }
I20260619 11:08:36.095837 677 sys_catalog.cc:564] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:08:36.096549 683 sys_catalog.cc:455] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: SysCatalogTable state changed. Reason: New leader d002ae89c80d412b97adeb4fd5ea8def. Latest consensus state: current_term: 3 leader_uuid: "d002ae89c80d412b97adeb4fd5ea8def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } } }
I20260619 11:08:36.096660 683 sys_catalog.cc:458] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:36.097319 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d002ae89c80d412b97adeb4fd5ea8def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d002ae89c80d412b97adeb4fd5ea8def" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41709 } } }
I20260619 11:08:36.097373 687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:08:36.097409 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:36.102098 687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:08:36.102962 687 catalog_manager.cc:1259] Loaded cluster ID: 4b00898aff444d619ca8a3607009454d
I20260619 11:08:36.103021 687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:08:36.105602 687 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:08:36.106253 687 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Loaded TSK: 0
I20260619 11:08:36.106331 687 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d002ae89c80d412b97adeb4fd5ea8def: Loaded TSK: 1
F20260619 11:08:36.107530 687 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f41c0d299ad google::LogMessage::Fail() at ??:0
@ 0x7f41c0d2c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f41c0d294de google::LogMessage::Flush() at ??:0
@ 0x7f41c0d2aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f41c39ed18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f41c39e5f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f41cb68e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f41cb66899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f41cb669178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f41cb6aadf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f41cb6c982b std::function<>::operator()() at ??:0
@ 0x7f41cb668b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f41cb669f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f41cb666579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f41cb6aa640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f41c2f7b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f41c2f7bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f41c2f7d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f41c2f6c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f41c45b06db start_thread at ??:0
@ 0x7f41bf82761f clone at ??:0
W20260619 11:08:36.207001 679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:36.207228 679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:36.207270 679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:36.211243 679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:36.211294 679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:36.211378 679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:08:36.216596 679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:32823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=39379
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:36.217636 679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:36.218600 679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:36.229115 728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:36.229153 726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:36.229153 725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:36.229328 679 server_base.cc:1034] running on GCE node
I20260619 11:08:36.229935 679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:36.230544 679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:36.231755 679 hybrid_clock.cc:648] HybridClock initialized: now 1781867316231722 us; error 47 us; skew 500 ppm
Jun 19 11:08:36 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:36.234591 679 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:08:36.235772 679 webserver.cc:460] Webserver started at http://127.16.61.1:39379/ using document root <none> and password file <none>
I20260619 11:08:36.236431 679 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:36.236532 679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:36.241432 679 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:08:36.244175 735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:36.245663 679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:08:36.245781 679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "e3d286a467d3430ea868fc472c405194"
format_stamp: "Formatted at 2026-06-19 11:07:53 on dist-test-slave-8vcz"
server_key: "51b826b986c9776294c65f0cfe68f03a"
server_key_iv: "7d2108fa56b66eb558bc37cb26604b38"
server_key_version: "encryptionkey@0"
I20260619 11:08:36.246447 679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:36.266673 679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:36.270335 679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:36.270553 679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:36.271263 679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:36.272601 742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 11:08:36.276587 679 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 11:08:36.276659 679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 11:08:36.276727 679 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 11:08:36.278853 679 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 11:08:36.278913 679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 11:08:36.279202 742 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Bootstrap starting.
I20260619 11:08:36.285302 742 log.cc:824] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:36.289394 679 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:32823
I20260619 11:08:36.289433 850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:32823 every 8 connection(s)
I20260619 11:08:36.290408 679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:08:36.293922 742 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 11:08:36.294356 742 tablet_bootstrap.cc:492] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Bootstrap complete.
I20260619 11:08:36.294668 742 ts_tablet_manager.cc:1397] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260619 11:08:36.295908 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 679
I20260619 11:08:36.296368 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:34723
--local_ip_for_outbound_sockets=127.16.61.2
--tserver_master_addrs=127.16.61.62:41709
--webserver_port=36923
--webserver_interface=127.16.61.2
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:08:36.297690 742 raft_consensus.cc:348] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:36.298213 742 raft_consensus.cc:729] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3d286a467d3430ea868fc472c405194, State: Initialized, Role: FOLLOWER
I20260619 11:08:36.298691 742 consensus_queue.cc:260] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:36.299536 742 ts_tablet_manager.cc:1428] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260619 11:08:36.331073 853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:08:36.292473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:49785)
0619 11:08:36.292980 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.293028 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:36.294010 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:36.329806 (+ 35796us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":131,"threads_started":1}
W20260619 11:08:36.331707 851 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:41709 (0 consecutive failures): Network error: Failed to ping master at 127.16.61.62:41709: Client connection negotiation failed: client connection to 127.16.61.62:41709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 11:08:36.332412 853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:36.332127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:58437)
0619 11:08:36.332265 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.332343 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 11:08:36.333112 853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:36.332865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:37343)
0619 11:08:36.332987 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.333038 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 11:08:36.333302 851 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:08:36.411286 854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:36.411507 854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:36.411578 854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:36.415488 854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:36.415539 854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:36.415629 854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:08:36.420548 854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:34723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=36923
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:36.421435 854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:36.422351 854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:36.431417 860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:36.431416 863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:36.431515 854 server_base.cc:1034] running on GCE node
W20260619 11:08:36.431411 861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:36.432014 854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:36.432605 854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:36.433784 854 hybrid_clock.cc:648] HybridClock initialized: now 1781867316433770 us; error 23 us; skew 500 ppm
Jun 19 11:08:36 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:36.436429 854 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:08:36.437206 854 webserver.cc:460] Webserver started at http://127.16.61.2:36923/ using document root <none> and password file <none>
I20260619 11:08:36.437719 854 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:36.437806 854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:36.441322 854 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:08:36.443156 870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:36.444267 854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:36.444393 854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "be16a9daefeb48cb91b0209a55ab042c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "a539a3511958a201280b4c66df216036"
server_key_iv: "dddc53a60cadf71848e93a15da5f5294"
server_key_version: "encryptionkey@0"
I20260619 11:08:36.444887 854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:36.455238 854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:36.458273 854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:36.458452 854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:36.458969 854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:36.459764 854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:36.459824 854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:36.459887 854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:36.459931 854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:36.468351 854 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:34723
I20260619 11:08:36.468393 983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:34723 every 8 connection(s)
I20260619 11:08:36.469261 854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:08:36.472715 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:36.471086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:46263)
0619 11:08:36.471497 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.471600 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
W20260619 11:08:36.473140 984 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:41709 (0 consecutive failures): Network error: Failed to ping master at 127.16.61.62:41709: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
I20260619 11:08:36.473368 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 854
I20260619 11:08:36.473694 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:36.473504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:49679)
0619 11:08:36.473593 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.473643 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 11:08:36.473793 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:44987
--local_ip_for_outbound_sockets=127.16.61.3
--tserver_master_addrs=127.16.61.62:41709
--webserver_port=41603
--webserver_interface=127.16.61.3
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:36031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:08:36.474337 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:36.474154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:60817)
0619 11:08:36.474242 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.474288 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 11:08:36.474563 984 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 11:08:36.579555 987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:36.579885 987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:36.579972 987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:36.583896 987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:36.583978 987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:36.584065 987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:08:36.589191 987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:44987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=41603
--tserver_master_addrs=127.16.61.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:36.590149 987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:36.591120 987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:36.600045 992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:36.600003 995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:36.600040 993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:36.600518 987 server_base.cc:1034] running on GCE node
I20260619 11:08:36.600888 987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:36.601459 987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:36.602641 987 hybrid_clock.cc:648] HybridClock initialized: now 1781867316602624 us; error 30 us; skew 500 ppm
Jun 19 11:08:36 dist-test-slave-8vcz krb5kdc[32001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:36.605209 987 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:08:36.606040 987 webserver.cc:460] Webserver started at http://127.16.61.3:41603/ using document root <none> and password file <none>
I20260619 11:08:36.606564 987 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:36.606658 987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:36.610217 987 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:36.612212 1002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:36.613440 987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:36.613579 987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c"
format_stamp: "Formatted at 2026-06-19 11:07:54 on dist-test-slave-8vcz"
server_key: "3360bc0f9dc8571a70bf9cf8471da5e3"
server_key_iv: "d022ed57370af37263ecae749df20b99"
server_key_version: "encryptionkey@0"
I20260619 11:08:36.614135 987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:36.631213 987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:36.633903 987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:36.634090 987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:36.634630 987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:36.635406 987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:36.635473 987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:36.635538 987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:36.635586 987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:36.644265 987 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:44987
I20260619 11:08:36.644289 1115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:44987 every 8 connection(s)
I20260619 11:08:36.645184 987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:08:36.648926 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:36.647193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:50043)
0619 11:08:36.647623 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.647763 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
W20260619 11:08:36.649428 1116 heartbeater.cc:644] Failed to heartbeat to 127.16.61.62:41709 (0 consecutive failures): Network error: Failed to ping master at 127.16.61.62:41709: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
I20260619 11:08:36.649541 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 987
I20260619 11:08:36.650053 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:36.649787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:59271)
0619 11:08:36.649917 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.649981 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 11:08:36.650765 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:36.650564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:42355)
0619 11:08:36.650662 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:36.650717 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 11:08:36.650921 1116 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 11:08:37.334295 1120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:37.333914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:38955)
0619 11:08:37.334145 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:37.334193 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260619 11:08:37.475395 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:37.475044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:33139)
0619 11:08:37.475255 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:37.475303 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
I20260619 11:08:37.651793 1122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:37.651384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:34605)
0619 11:08:37.651605 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:37.651685 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260619 11:08:38.152330 1123 raft_consensus.cc:482] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:38.152576 1123 raft_consensus.cc:504] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:38.153630 1123 leader_election.cc:288] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987), be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723)
Jun 19 11:08:38 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.2@KRBTEST.COM
Jun 19 11:08:38 dist-test-slave-8vcz krb5kdc[32001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.3@KRBTEST.COM
I20260619 11:08:38.167212 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:38.153997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:36531 (local address 127.16.61.2:34723)
0619 11:08:38.154381 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:38.154390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:38.154419 (+ 29us) server_negotiation.cc:407] Connection header received
0619 11:08:38.155296 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:38.155302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:38.155609 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:38.155942 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:38.157171 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:38.158113 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:38.159296 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:38.159790 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:38.159948 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:38.162524 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:38.162530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:38.162540 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:38.162567 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:38.164561 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:38.165249 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:38.165255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:38.165260 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:38.165414 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:38.165806 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:38.165811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:38.165813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:38.166188 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:38.166316 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:38.166804 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 11:08:38.167046 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":81,"threads_started":1}
I20260619 11:08:38.167212 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:38.153826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:34723 (local address 127.16.61.1:36531)
0619 11:08:38.154114 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:38.154129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:38.154224 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:38.155991 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:38.155997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:38.156549 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:38.157017 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:38.157034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:38.158258 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:38.158261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:38.159182 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:38.159190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:38.159903 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:38.159907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:38.160131 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:38.160764 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:38.160779 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:38.162320 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:38.164676 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:38.164680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:38.164690 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:38.165112 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:38.165521 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:38.165524 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:38.165526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:38.165694 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:38.166307 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:38.166313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:38.166534 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 11:08:38.166698 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 11:08:38.167012 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":133,"spinlock_wait_cycles":27136,"thread_start_us":68,"threads_started":1}
I20260619 11:08:38.167266 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:38.153941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:46833 (local address 127.16.61.3:44987)
0619 11:08:38.154353 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:38.154365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:38.154399 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:08:38.155267 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:38.155275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:38.155609 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:38.155943 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:38.157182 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:38.158104 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:38.159301 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:38.159795 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:38.159948 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:38.162831 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:38.162840 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:38.162857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:38.162891 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:38.164747 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:38.165252 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:38.165256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:38.165261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:38.165414 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:38.165858 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:38.165860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:38.165861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:38.166199 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:38.166328 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:38.166819 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 11:08:38.167081 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":72,"threads_started":1}
I20260619 11:08:38.167275 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:38.153783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:44987 (local address 127.16.61.1:46833)
0619 11:08:38.154082 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:38.154097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:38.154199 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:38.156024 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:38.156027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:38.156557 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:38.157017 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:38.157031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:38.158249 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:38.158252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:38.159181 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:38.159191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:38.159917 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:38.159921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:38.160133 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:38.160815 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:38.160827 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:38.162651 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:38.164835 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:38.164838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:38.164840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:38.165158 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:38.165685 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:38.165688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:38.165690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:38.165774 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:38.166316 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:38.166319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:38.166520 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 11:08:38.166698 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 11:08:38.167100 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":301,"spinlock_wait_cycles":27264,"thread_start_us":64,"threads_started":1}
W20260619 11:08:38.169617 736 leader_election.cc:341] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723): Not found: Tablet not found: 157ef5b6244f484193a7ba4c505fa3e6
W20260619 11:08:38.169766 738 leader_election.cc:341] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987): Not found: Tablet not found: 157ef5b6244f484193a7ba4c505fa3e6
I20260619 11:08:38.169837 738 leader_election.cc:302] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d286a467d3430ea868fc472c405194; no voters: 9ed8292ccf5b4c1db50fa2d55e30b57c, be16a9daefeb48cb91b0209a55ab042c
I20260619 11:08:38.170099 1123 raft_consensus.cc:2726] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:08:38.335218 1124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:38.334963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:54421)
0619 11:08:38.335089 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:38.335138 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 11:08:38.476387 1128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:38.476013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:46499)
0619 11:08:38.476239 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:38.476288 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260619 11:08:38.652810 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:38.652420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:53671)
0619 11:08:38.652668 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:38.652715 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260619 11:08:39.336277 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:39.335885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:57911)
0619 11:08:39.336127 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:39.336181 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260619 11:08:39.477380 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:39.477018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:47809)
0619 11:08:39.477252 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:39.477300 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260619 11:08:39.653816 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:39.653460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:57807)
0619 11:08:39.653685 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:39.653736 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260619 11:08:40.337353 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:40.336976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:35127)
0619 11:08:40.337206 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:40.337257 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260619 11:08:40.348943 1134 raft_consensus.cc:482] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:40.349056 1134 raft_consensus.cc:504] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:40.349512 1134 leader_election.cc:288] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987), be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723)
W20260619 11:08:40.350231 738 leader_election.cc:341] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987): Not found: Tablet not found: 157ef5b6244f484193a7ba4c505fa3e6
W20260619 11:08:40.350317 736 leader_election.cc:341] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723): Not found: Tablet not found: 157ef5b6244f484193a7ba4c505fa3e6
I20260619 11:08:40.350419 736 leader_election.cc:302] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d286a467d3430ea868fc472c405194; no voters: 9ed8292ccf5b4c1db50fa2d55e30b57c, be16a9daefeb48cb91b0209a55ab042c
I20260619 11:08:40.350581 1134 raft_consensus.cc:2726] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:08:40.478308 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:40.477976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:52385)
0619 11:08:40.478179 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:40.478229 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260619 11:08:40.654899 1136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:40.654493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:59591)
0619 11:08:40.654734 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:40.654803 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260619 11:08:41.338798 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:41.338378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:57599)
0619 11:08:41.338651 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:41.338703 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":70,"threads_started":1}
I20260619 11:08:41.479290 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:41.478928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:53935)
0619 11:08:41.479155 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:41.479206 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":70,"threads_started":1}
I20260619 11:08:41.656225 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:41.655717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:34141)
0619 11:08:41.656006 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:41.656099 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":72,"threads_started":1}
I20260619 11:08:42.055274 1140 raft_consensus.cc:482] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:42.055394 1140 raft_consensus.cc:504] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:42.055878 1140 leader_election.cc:288] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987), be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723)
W20260619 11:08:42.056633 736 leader_election.cc:341] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723): Not found: Tablet not found: 157ef5b6244f484193a7ba4c505fa3e6
W20260619 11:08:42.056746 738 leader_election.cc:341] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987): Not found: Tablet not found: 157ef5b6244f484193a7ba4c505fa3e6
I20260619 11:08:42.056851 738 leader_election.cc:302] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d286a467d3430ea868fc472c405194; no voters: 9ed8292ccf5b4c1db50fa2d55e30b57c, be16a9daefeb48cb91b0209a55ab042c
I20260619 11:08:42.057001 1140 raft_consensus.cc:2726] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:08:42.339965 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:42.339526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:53055)
0619 11:08:42.339818 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:42.339871 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":126,"threads_started":1}
I20260619 11:08:42.480330 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:42.479981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:39081)
0619 11:08:42.480197 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:42.480244 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
I20260619 11:08:42.657254 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:42.656859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:36723)
0619 11:08:42.657121 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:42.657173 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260619 11:08:43.341077 1144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:43.340689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:33221)
0619 11:08:43.340929 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:43.340982 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260619 11:08:43.481420 1145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:43.481015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:38867)
0619 11:08:43.481283 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:43.481336 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":68,"threads_started":1}
I20260619 11:08:43.658375 1146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:43.657912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:53533)
0619 11:08:43.658236 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:43.658289 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":74,"threads_started":1}
I20260619 11:08:44.342160 1147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:44.341759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:40461)
0619 11:08:44.342011 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:44.342063 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20260619 11:08:44.482543 1148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:44.482139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:54347)
0619 11:08:44.482394 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:44.482453 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260619 11:08:44.659394 1149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:44.659025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:53819)
0619 11:08:44.659263 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:44.659311 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260619 11:08:45.343266 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:45.342853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:52555)
0619 11:08:45.343116 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:45.343170 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260619 11:08:45.483572 1151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:45.483202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:53693)
0619 11:08:45.483431 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:45.483484 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260619 11:08:45.660578 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:45.660159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:47779)
0619 11:08:45.660427 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:45.660481 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260619 11:08:45.851742 1153 raft_consensus.cc:482] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:45.851859 1153 raft_consensus.cc:504] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed8292ccf5b4c1db50fa2d55e30b57c" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44987 } } peers { permanent_uuid: "e3d286a467d3430ea868fc472c405194" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 32823 } } peers { permanent_uuid: "be16a9daefeb48cb91b0209a55ab042c" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 34723 } }
I20260619 11:08:45.852326 1153 leader_election.cc:288] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987), be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723)
W20260619 11:08:45.852991 736 leader_election.cc:341] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be16a9daefeb48cb91b0209a55ab042c (127.16.61.2:34723): Not found: Tablet not found: 157ef5b6244f484193a7ba4c505fa3e6
W20260619 11:08:45.853140 738 leader_election.cc:341] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ed8292ccf5b4c1db50fa2d55e30b57c (127.16.61.3:44987): Not found: Tablet not found: 157ef5b6244f484193a7ba4c505fa3e6
I20260619 11:08:45.853223 738 leader_election.cc:302] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3d286a467d3430ea868fc472c405194; no voters: 9ed8292ccf5b4c1db50fa2d55e30b57c, be16a9daefeb48cb91b0209a55ab042c
I20260619 11:08:45.853364 1153 raft_consensus.cc:2726] T 157ef5b6244f484193a7ba4c505fa3e6 P e3d286a467d3430ea868fc472c405194 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 11:08:46.344484 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:46.344101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:46733)
0619 11:08:46.344341 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:46.344390 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260619 11:08:46.484651 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:46.484274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:43637)
0619 11:08:46.484505 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:46.484555 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
I20260619 11:08:46.661645 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:46.661285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:57835)
0619 11:08:46.661506 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:46.661558 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260619 11:08:47.345613 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:47.345168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:34231)
0619 11:08:47.345447 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:47.345514 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260619 11:08:47.485740 1158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:47.485361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:45515)
0619 11:08:47.485596 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:47.485650 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260619 11:08:47.662690 1159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:47.662318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:35027)
0619 11:08:47.662547 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:47.662602 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260619 11:08:48.346940 1160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:48.346508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:37633)
0619 11:08:48.346772 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:48.346839 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260619 11:08:48.486871 1161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:48.486475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:34433)
0619 11:08:48.486721 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:48.486781 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":69,"threads_started":1}
I20260619 11:08:48.663924 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:48.663483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:50317)
0619 11:08:48.663774 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:48.663834 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":71,"threads_started":1}
I20260619 11:08:49.348199 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:49.347785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:46147)
0619 11:08:49.348038 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:49.348091 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
I20260619 11:08:49.487907 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:49.487508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:44355)
0619 11:08:49.487767 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:49.487818 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":66,"threads_started":1}
I20260619 11:08:49.664981 1165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:49.664616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:50393)
0619 11:08:49.664840 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:49.664893 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260619 11:08:50.349301 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:50.348882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:43225)
0619 11:08:50.349156 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:50.349209 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":102,"threads_started":1}
I20260619 11:08:50.488941 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:50.488548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:60285)
0619 11:08:50.488796 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:50.488848 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260619 11:08:50.666003 1168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:50.665647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:39045)
0619 11:08:50.665864 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:50.665914 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
I20260619 11:08:51.350538 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:51.350129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.1:49993)
0619 11:08:51.350390 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:51.350444 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260619 11:08:51.490265 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:51.489797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.2:45153)
0619 11:08:51.490059 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:51.490126 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":94,"threads_started":1}
I20260619 11:08:51.657624 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 679
I20260619 11:08:51.663378 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 854
I20260619 11:08:51.667258 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:51.666876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41709 (local address 127.16.61.3:40655)
0619 11:08:51.667116 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:51.667170 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.62:41709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20260619 11:08:51.668850 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 987
2026-06-19T11:08:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59260 ms)
[----------] 30 tests from SecurityITest (236986 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:08:51 dist-test-slave-8vcz krb5kdc[1174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:08:51 dist-test-slave-8vcz krb5kdc[1174](info): set up 2 sockets
Jun 19 11:08:51 dist-test-slave-8vcz krb5kdc[1174](info): commencing operation
krb5kdc: starting...
W20260619 11:08:52.711439 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:08:52 dist-test-slave-8vcz krb5kdc[1174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:08:52.826710 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:37159
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:37159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:52.932313 1187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:52.932569 1187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:52.932639 1187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:52.936514 1187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:08:52.936592 1187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:52.936630 1187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:52.936687 1187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:08:52.936728 1187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:08:52.941793 1187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:37159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:52.942718 1187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:52.943645 1187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:52.951921 1192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:52.951962 1193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:52.951942 1187 server_base.cc:1034] running on GCE node
W20260619 11:08:52.951932 1195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:52.952708 1187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:52.953715 1187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:52.954905 1187 hybrid_clock.cc:648] HybridClock initialized: now 1781867332954868 us; error 56 us; skew 500 ppm
Jun 19 11:08:52 dist-test-slave-8vcz krb5kdc[1174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867332, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:52.957556 1187 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:08:52.958292 1187 webserver.cc:460] Webserver started at http://127.16.61.62:45491/ using document root <none> and password file <none>
I20260619 11:08:52.958767 1187 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:52.958823 1187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:52.958967 1187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:52.960850 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "13cf963db2a5419aaa6fcc1f66c509a1"
format_stamp: "Formatted at 2026-06-19 11:08:52 on dist-test-slave-8vcz"
server_key: "2e90885e95c3d68a47dd213f2b3331d8"
server_key_iv: "1d3bbe17e5d970d79a51a79c485d85e0"
server_key_version: "encryptionkey@0"
I20260619 11:08:52.961392 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "13cf963db2a5419aaa6fcc1f66c509a1"
format_stamp: "Formatted at 2026-06-19 11:08:52 on dist-test-slave-8vcz"
server_key: "2e90885e95c3d68a47dd213f2b3331d8"
server_key_iv: "1d3bbe17e5d970d79a51a79c485d85e0"
server_key_version: "encryptionkey@0"
I20260619 11:08:52.964802 1187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:08:52.966953 1202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:52.968176 1187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:52.968271 1187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "13cf963db2a5419aaa6fcc1f66c509a1"
format_stamp: "Formatted at 2026-06-19 11:08:52 on dist-test-slave-8vcz"
server_key: "2e90885e95c3d68a47dd213f2b3331d8"
server_key_iv: "1d3bbe17e5d970d79a51a79c485d85e0"
server_key_version: "encryptionkey@0"
I20260619 11:08:52.968349 1187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:52.993425 1187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:52.996182 1187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:52.996323 1187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:53.002894 1187 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:37159
I20260619 11:08:53.002923 1254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:37159 every 8 connection(s)
I20260619 11:08:53.003844 1187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:08:53.006345 1255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:53.011402 1255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1: Bootstrap starting.
I20260619 11:08:53.012071 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 1187
I20260619 11:08:53.012163 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:08:53.012444 16628 external_mini_cluster.cc:1428] Setting key 04baa274bfe9fca06df70b1501191bf2
I20260619 11:08:53.014017 1255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:53.014706 1255 log.cc:824] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:53.016716 1255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1: No bootstrap required, opened a new log
Jun 19 11:08:53 dist-test-slave-8vcz krb5kdc[1174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:53.020959 1255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf963db2a5419aaa6fcc1f66c509a1" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37159 } }
I20260619 11:08:53.021180 1255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:53.021250 1255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13cf963db2a5419aaa6fcc1f66c509a1, State: Initialized, Role: FOLLOWER
I20260619 11:08:53.021762 1255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf963db2a5419aaa6fcc1f66c509a1" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37159 } }
I20260619 11:08:53.021934 1255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:08:53.022008 1255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:08:53.022125 1255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:53.023231 1255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf963db2a5419aaa6fcc1f66c509a1" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37159 } }
I20260619 11:08:53.023583 1255 leader_election.cc:302] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13cf963db2a5419aaa6fcc1f66c509a1; no voters:
I20260619 11:08:53.023919 1255 leader_election.cc:288] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:08:53.024019 1260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:53.024247 1260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [term 1 LEADER]: Becoming Leader. State: Replica: 13cf963db2a5419aaa6fcc1f66c509a1, State: Running, Role: LEADER
I20260619 11:08:53.024555 1260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf963db2a5419aaa6fcc1f66c509a1" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37159 } }
I20260619 11:08:53.025133 1255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:08:53.026126 1262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13cf963db2a5419aaa6fcc1f66c509a1. Latest consensus state: current_term: 1 leader_uuid: "13cf963db2a5419aaa6fcc1f66c509a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf963db2a5419aaa6fcc1f66c509a1" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37159 } } }
I20260619 11:08:53.026243 1262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:53.027124 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.013538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.16.61.62:37159)
0619 11:08:53.013932 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.013946 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.013978 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:08:53.014618 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.014633 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.014980 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.015339 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:53.016171 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.017189 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.018363 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.018816 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.018935 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.021717 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:53.021723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:53.021737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:53.021765 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:53.023764 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.024331 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.024337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.024343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.024475 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.024864 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.024868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.024870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.025329 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:53.025507 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.025842 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.026184 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":108,"threads_started":1}
I20260619 11:08:53.027516 1261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13cf963db2a5419aaa6fcc1f66c509a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13cf963db2a5419aaa6fcc1f66c509a1" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37159 } } }
I20260619 11:08:53.027705 1261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1 [sys.catalog]: This master's current role is: LEADER
W20260619 11:08:53.030257 1275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:08:53.030320 1275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:08:53.030400 1269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:08:53.031131 1269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:08:53.036149 1269 catalog_manager.cc:1347] Generated new cluster ID: 495f5f6edc3d4f7babedb54ce32af26d
I20260619 11:08:53.036212 1269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:08:53.050169 1269 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:08:53.051020 1269 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:08:53.057464 1269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1: Generated new TSK 0
I20260619 11:08:53.058274 1269 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:08:53.088828 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:37159
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:53.197170 1281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:53.197396 1281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:53.197436 1281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:53.201335 1281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:53.201386 1281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:53.201442 1281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:08:53.206456 1281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:53.207317 1281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:53.208338 1281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:53.217232 1289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:53.217252 1286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:53.217232 1287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:53.217373 1281 server_base.cc:1034] running on GCE node
I20260619 11:08:53.217905 1281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:53.218472 1281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:53.219631 1281 hybrid_clock.cc:648] HybridClock initialized: now 1781867333219619 us; error 23 us; skew 500 ppm
Jun 19 11:08:53 dist-test-slave-8vcz krb5kdc[1174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867333, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:53.222252 1281 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:08:53.223004 1281 webserver.cc:460] Webserver started at http://127.16.61.1:43569/ using document root <none> and password file <none>
I20260619 11:08:53.223498 1281 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:53.223555 1281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:53.223778 1281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:53.225678 1281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "20916f537d0b4a708caa14a2baf2d1cf"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "0e57df097e9062a1e414ecf48c63b46c"
server_key_iv: "ba9ddd4ebbe3cb920565fd3a5da22efb"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.226284 1281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "20916f537d0b4a708caa14a2baf2d1cf"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "0e57df097e9062a1e414ecf48c63b46c"
server_key_iv: "ba9ddd4ebbe3cb920565fd3a5da22efb"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.229840 1281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:08:53.232159 1296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.233286 1281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:53.233422 1281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "20916f537d0b4a708caa14a2baf2d1cf"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "0e57df097e9062a1e414ecf48c63b46c"
server_key_iv: "ba9ddd4ebbe3cb920565fd3a5da22efb"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.233530 1281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:53.258865 1281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:53.262003 1281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:53.262202 1281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:53.262758 1281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:53.263530 1281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:53.263599 1281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.263705 1281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:53.263764 1281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.272797 1281 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:33741
I20260619 11:08:53.272833 1409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:33741 every 8 connection(s)
I20260619 11:08:53.273641 1281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:08:53.274605 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 1281
I20260619 11:08:53.274746 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:08:53.275020 16628 external_mini_cluster.cc:1428] Setting key 247df52354ba488bce3ec6dea6499e46
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
Jun 19 11:08:53 dist-test-slave-8vcz krb5kdc[1174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867333, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
I20260619 11:08:53.288105 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.275491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:47635 (local address 127.16.61.62:37159)
0619 11:08:53.275632 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.275637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.276640 (+ 1003us) server_negotiation.cc:407] Connection header received
0619 11:08:53.278582 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.278585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.278639 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.278716 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:53.279885 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.280481 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.281612 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.281861 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.281969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.284366 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:53.284370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:53.284372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:53.284398 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:53.285792 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.286275 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.286279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.286283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.286370 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.286765 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.286768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.286769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.286948 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:53.287062 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.287707 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.287849 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:08:53.288811 1412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.275943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:37159 (local address 127.16.61.1:47635)
0619 11:08:53.276476 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.276532 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:53.278354 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:53.278889 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:53.278896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:53.279232 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:53.279749 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.279765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.280615 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.280619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.281488 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.281497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.282016 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.282020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.282189 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.282772 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:53.282788 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:53.284205 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:53.285903 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:53.285908 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:53.285921 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:53.286164 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:53.286463 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:53.286466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:53.286468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:53.286641 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:53.287087 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:53.287096 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:53.287409 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 11:08:53.287611 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 11:08:53.287995 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":151,"threads_started":1}
I20260619 11:08:53.290040 1410 heartbeater.cc:347] Connected to a master server at 127.16.61.62:37159
I20260619 11:08:53.290307 1410 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:53.290890 1410 heartbeater.cc:511] Master 127.16.61.62:37159 requested a full tablet report, sending...
I20260619 11:08:53.292464 1219 ts_manager.cc:194] Registered new tserver with Master: 20916f537d0b4a708caa14a2baf2d1cf (127.16.61.1:33741)
I20260619 11:08:53.293541 1219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:47635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:08:53.302542 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:37159
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:53.408264 1415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:53.408490 1415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:53.408530 1415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:53.412526 1415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:53.412576 1415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:53.412662 1415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:08:53.417676 1415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:53.418589 1415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:53.419523 1415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:08:53.428660 1415 server_base.cc:1034] running on GCE node
W20260619 11:08:53.428675 1421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:53.428684 1420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:53.428869 1423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:53.429212 1415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:53.429817 1415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:53.431011 1415 hybrid_clock.cc:648] HybridClock initialized: now 1781867333431003 us; error 27 us; skew 500 ppm
Jun 19 11:08:53 dist-test-slave-8vcz krb5kdc[1174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867333, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:53.433715 1415 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:08:53.434514 1415 webserver.cc:460] Webserver started at http://127.16.61.2:33453/ using document root <none> and password file <none>
I20260619 11:08:53.435034 1415 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:53.435120 1415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:53.435341 1415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:53.437294 1415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "57786d30a9074ec88bee47a0562f4810"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "d4544f9ea443be965014697edcca53d5"
server_key_iv: "7e7dae7f98d24f1e051e35d0cc6fe7db"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.437822 1415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "57786d30a9074ec88bee47a0562f4810"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "d4544f9ea443be965014697edcca53d5"
server_key_iv: "7e7dae7f98d24f1e051e35d0cc6fe7db"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.441283 1415 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:08:53.443459 1430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.444597 1415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:53.444725 1415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "57786d30a9074ec88bee47a0562f4810"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "d4544f9ea443be965014697edcca53d5"
server_key_iv: "7e7dae7f98d24f1e051e35d0cc6fe7db"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.444826 1415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:53.467615 1415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:53.470707 1415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:53.470844 1415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:53.471318 1415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:53.472141 1415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:53.472193 1415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.472231 1415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:53.472247 1415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.480607 1415 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:37917
I20260619 11:08:53.480638 1543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:37917 every 8 connection(s)
I20260619 11:08:53.481443 1415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:08:53.487920 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 1415
I20260619 11:08:53.488092 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:08:53.488389 16628 external_mini_cluster.cc:1428] Setting key fe7e65b48e6994bc7a3e4354f6e079ff
Jun 19 11:08:53 dist-test-slave-8vcz krb5kdc[1174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867333, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:53.494617 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.483132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:56683 (local address 127.16.61.62:37159)
0619 11:08:53.483279 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.483283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.484053 (+ 770us) server_negotiation.cc:407] Connection header received
0619 11:08:53.485353 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.485356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.485415 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.485512 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:53.486575 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.487099 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.488440 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.488691 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.488803 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.491178 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:53.491182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:53.491184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:53.491214 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:53.492640 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.493136 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.493141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.493143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.493224 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.493557 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.493562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.493564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.493749 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:53.493858 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.494322 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.494453 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 11:08:53.495355 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.483463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:37159 (local address 127.16.61.2:56683)
0619 11:08:53.483916 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.483954 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:53.485150 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:53.485679 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:53.485685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:53.486011 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:53.486439 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.486452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.487231 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.487236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.488325 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.488334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.488836 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.488840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.489025 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.489609 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:53.489631 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:53.491020 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:53.492766 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:53.492771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:53.492784 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:53.493033 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:53.493325 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:53.493328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:53.493330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:53.493463 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:53.493854 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:53.493861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:53.494093 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 11:08:53.494284 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 11:08:53.494586 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":163,"threads_started":1}
I20260619 11:08:53.496614 1544 heartbeater.cc:347] Connected to a master server at 127.16.61.62:37159
I20260619 11:08:53.496840 1544 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:53.497298 1544 heartbeater.cc:511] Master 127.16.61.62:37159 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
I20260619 11:08:53.498399 1219 ts_manager.cc:194] Registered new tserver with Master: 57786d30a9074ec88bee47a0562f4810 (127.16.61.2:37917)
I20260619 11:08:53.498958 1219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:56683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:08:53.514909 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:37159
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:53.621193 1549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:53.621420 1549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:53.621459 1549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:53.625434 1549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:53.625483 1549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:53.625537 1549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:08:53.630666 1549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:53.631523 1549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:53.632398 1549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:53.641669 1557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:53.641682 1554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:53.641702 1555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:53.641881 1549 server_base.cc:1034] running on GCE node
I20260619 11:08:53.642334 1549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:53.642944 1549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:53.644125 1549 hybrid_clock.cc:648] HybridClock initialized: now 1781867333644114 us; error 30 us; skew 500 ppm
Jun 19 11:08:53 dist-test-slave-8vcz krb5kdc[1174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867333, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:53.646641 1549 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:08:53.647439 1549 webserver.cc:460] Webserver started at http://127.16.61.3:40057/ using document root <none> and password file <none>
I20260619 11:08:53.648051 1549 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:53.648138 1549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:53.648379 1549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:53.650408 1549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "0d9532f2b7e85d03699597bb0e8002ce"
server_key_iv: "740ebaaab2db63d13ceb7d87ac9cd776"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.650954 1549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "0d9532f2b7e85d03699597bb0e8002ce"
server_key_iv: "740ebaaab2db63d13ceb7d87ac9cd776"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.654417 1549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:08:53.656721 1564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.657895 1549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:08:53.658032 1549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "0d9532f2b7e85d03699597bb0e8002ce"
server_key_iv: "740ebaaab2db63d13ceb7d87ac9cd776"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.658138 1549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:53.670495 1549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:53.673652 1549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:53.673851 1549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:53.674381 1549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:53.675132 1549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:53.675201 1549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.675266 1549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:53.675307 1549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:53.683908 1549 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:37425
I20260619 11:08:53.683979 1677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:37425 every 8 connection(s)
I20260619 11:08:53.684763 1549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:08:53.690171 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 1549
I20260619 11:08:53.690301 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:08:53.690575 16628 external_mini_cluster.cc:1428] Setting key 27bf18d89dc2772943bfbd9124aa28e4
Jun 19 11:08:53 dist-test-slave-8vcz krb5kdc[1174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867333, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:53.697907 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.686402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:48019 (local address 127.16.61.62:37159)
0619 11:08:53.686545 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.686548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.687278 (+ 730us) server_negotiation.cc:407] Connection header received
0619 11:08:53.688674 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.688678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.688744 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.688835 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:53.689954 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.690580 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.692065 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.692394 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.692504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.694771 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:53.694774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:53.694776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:53.694816 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:53.696258 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.696691 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.696693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.696694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.696775 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.697045 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.697046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.697048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.697200 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:53.697275 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.697692 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.697794 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:08:53.698616 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.686727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:37159 (local address 127.16.61.3:48019)
0619 11:08:53.687135 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.687174 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:53.688433 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:53.688996 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:53.689003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:53.689361 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:53.689800 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.689814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.690742 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.690745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.691959 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.691967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.692556 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.692560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.692704 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.693193 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:53.693208 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:53.694549 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:53.696354 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:53.696358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:53.696367 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:53.696606 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:53.696851 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:53.696853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:53.696854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:53.696970 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:53.697286 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:53.697291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:53.697488 (+ 197us) client_negotiation.cc:769] Sending connection context
0619 11:08:53.697648 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 11:08:53.697867 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":112,"threads_started":1}
I20260619 11:08:53.699558 1678 heartbeater.cc:347] Connected to a master server at 127.16.61.62:37159
I20260619 11:08:53.699795 1678 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:53.700237 1678 heartbeater.cc:511] Master 127.16.61.62:37159 requested a full tablet report, sending...
I20260619 11:08:53.701148 1219 ts_manager.cc:194] Registered new tserver with Master: c1b86e1cbba44c8f8ec439a79ba51c0b (127.16.61.3:37425)
I20260619 11:08:53.701694 1219 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:48019
I20260619 11:08:53.704147 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:08:53.712796 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.705901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.16.61.62:37159)
0619 11:08:53.706056 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.706059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.706149 (+ 90us) server_negotiation.cc:407] Connection header received
0619 11:08:53.706248 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.706250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.706283 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.706345 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:53.706951 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.707427 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.708294 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.708498 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.708584 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.709862 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:53.709865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:53.709866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:53.709894 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:53.711170 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.711592 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.711595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.711596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.711718 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:53.711947 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:53.711949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:53.711950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:53.712188 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:53.712325 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.712495 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.712583 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 11:08:53.715546 1219 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:08:53.718518 1219 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:08:53.731586 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.726489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33741 (local address 127.0.0.1:43146)
0619 11:08:53.726803 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.726846 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:53.726960 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:53.727646 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:53.727652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:53.727703 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:53.727829 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.727837 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.729002 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.729005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.730004 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.730012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.731082 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.731086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.731197 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.731236 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 11:08:53.731345 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 11:08:53.731448 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260619 11:08:53.731645 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.726602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.16.61.1:33741)
0619 11:08:53.727064 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.727073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.727093 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:08:53.727195 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.727200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.727457 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.727609 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:53.727957 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.728893 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.730122 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.730947 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.731073 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.731207 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.731392 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.731492 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":167,"threads_started":1}
I20260619 11:08:53.732743 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.727160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:37425 (local address 127.0.0.1:39872)
0619 11:08:53.727421 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.727434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:53.727524 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:53.728707 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:53.728710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:53.728735 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:53.728825 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.728830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.730193 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.730196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.731346 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.731358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.732381 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.732385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.732469 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.732491 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:08:53.732560 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 11:08:53.732626 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":113,"threads_started":1}
I20260619 11:08:53.732844 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.727377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.16.61.3:37425)
0619 11:08:53.728377 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.728385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.728402 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:08:53.728458 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.728461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.728580 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.728689 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:53.728947 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.730082 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.731482 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.732276 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.732382 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.732509 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.732598 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.732682 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":139,"threads_started":1}
I20260619 11:08:53.733510 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.727502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:37917 (local address 127.0.0.1:51926)
0619 11:08:53.728046 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.728059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:53.728154 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:53.729375 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:53.729378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:53.729397 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:53.729480 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.729486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.730785 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.730788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.731943 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.731957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.733128 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.733132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.733233 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.733261 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:08:53.733336 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 11:08:53.733398 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":74,"threads_started":1}
I20260619 11:08:53.733577 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.728013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.16.61.2:37917)
0619 11:08:53.728955 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.728970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.728990 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:08:53.729058 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.729063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.729235 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.729374 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:53.729598 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.730630 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.732238 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.732978 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.733109 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.733239 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.733357 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.733438 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":135,"threads_started":1}
I20260619 11:08:53.734869 1344 tablet_service.cc:1467] Processing CreateTablet for tablet 8b37c784b31540ea975c7108a058c892 (DEFAULT_TABLE table=test-table [id=6ce3ea204a80422c8e0582b7296f003f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:53.735898 1344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b37c784b31540ea975c7108a058c892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:53.735496 1612 tablet_service.cc:1467] Processing CreateTablet for tablet 8b37c784b31540ea975c7108a058c892 (DEFAULT_TABLE table=test-table [id=6ce3ea204a80422c8e0582b7296f003f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:53.736498 1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b37c784b31540ea975c7108a058c892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:53.736501 1478 tablet_service.cc:1467] Processing CreateTablet for tablet 8b37c784b31540ea975c7108a058c892 (DEFAULT_TABLE table=test-table [id=6ce3ea204a80422c8e0582b7296f003f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:53.737414 1478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b37c784b31540ea975c7108a058c892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:53.742024 1695 tablet_bootstrap.cc:492] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf: Bootstrap starting.
I20260619 11:08:53.743484 1696 tablet_bootstrap.cc:492] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: Bootstrap starting.
I20260619 11:08:53.745092 1695 tablet_bootstrap.cc:654] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:53.745528 1696 tablet_bootstrap.cc:654] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:53.746474 1695 log.cc:824] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:53.746733 1696 log.cc:824] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:53.747139 1697 tablet_bootstrap.cc:492] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810: Bootstrap starting.
I20260619 11:08:53.748557 1696 tablet_bootstrap.cc:492] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: No bootstrap required, opened a new log
I20260619 11:08:53.748572 1695 tablet_bootstrap.cc:492] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf: No bootstrap required, opened a new log
I20260619 11:08:53.748762 1696 ts_tablet_manager.cc:1397] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:08:53.748751 1695 ts_tablet_manager.cc:1397] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 11:08:53.749281 1697 tablet_bootstrap.cc:654] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:53.750239 1697 log.cc:824] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:53.751860 1697 tablet_bootstrap.cc:492] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810: No bootstrap required, opened a new log
I20260619 11:08:53.752040 1697 ts_tablet_manager.cc:1397] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:08:53.752655 1695 raft_consensus.cc:348] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
I20260619 11:08:53.752655 1696 raft_consensus.cc:348] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
I20260619 11:08:53.752918 1695 raft_consensus.cc:374] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:53.752918 1696 raft_consensus.cc:374] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:53.752985 1696 raft_consensus.cc:729] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b86e1cbba44c8f8ec439a79ba51c0b, State: Initialized, Role: FOLLOWER
I20260619 11:08:53.753010 1695 raft_consensus.cc:729] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20916f537d0b4a708caa14a2baf2d1cf, State: Initialized, Role: FOLLOWER
I20260619 11:08:53.753463 1696 consensus_queue.cc:260] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
I20260619 11:08:53.753463 1695 consensus_queue.cc:260] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
I20260619 11:08:53.754305 1678 heartbeater.cc:503] Master 127.16.61.62:37159 was elected leader, sending a full tablet report...
I20260619 11:08:53.754313 1410 heartbeater.cc:503] Master 127.16.61.62:37159 was elected leader, sending a full tablet report...
I20260619 11:08:53.754436 1695 ts_tablet_manager.cc:1428] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:08:53.754436 1696 ts_tablet_manager.cc:1428] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:08:53.756764 1697 raft_consensus.cc:348] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
I20260619 11:08:53.757038 1697 raft_consensus.cc:374] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:53.757171 1697 raft_consensus.cc:729] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57786d30a9074ec88bee47a0562f4810, State: Initialized, Role: FOLLOWER
I20260619 11:08:53.758059 1697 consensus_queue.cc:260] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
I20260619 11:08:53.758632 1544 heartbeater.cc:503] Master 127.16.61.62:37159 was elected leader, sending a full tablet report...
I20260619 11:08:53.759086 1697 ts_tablet_manager.cc:1428] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 11:08:53.760667 1702 raft_consensus.cc:482] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:53.760830 1702 raft_consensus.cc:504] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
I20260619 11:08:53.761950 1702 leader_election.cc:288] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20916f537d0b4a708caa14a2baf2d1cf (127.16.61.1:33741), 57786d30a9074ec88bee47a0562f4810 (127.16.61.2:37917)
I20260619 11:08:53.765877 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.762141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33741 (local address 127.16.61.3:51585)
0619 11:08:53.762282 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.762303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:53.762370 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:53.762744 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:53.762746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:53.762760 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:53.762894 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.762903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.763775 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.763778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.764661 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.764670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.765467 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.765471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.765575 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.765596 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:08:53.765692 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 11:08:53.765756 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":19}
I20260619 11:08:53.765877 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.762220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:51585 (local address 127.16.61.1:33741)
0619 11:08:53.762361 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.762368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.762385 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:08:53.762489 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.762494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.762549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.762632 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:53.762989 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.763624 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.764789 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.765343 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.765467 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.765534 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.765704 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.765761 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:08:53.766074 1694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.762518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:35745 (local address 127.16.61.2:37917)
0619 11:08:53.762643 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.762647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.762875 (+ 228us) server_negotiation.cc:407] Connection header received
0619 11:08:53.762962 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.762968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.763011 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.763077 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:53.763335 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.763900 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.764995 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.765563 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.765686 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.765742 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.765904 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.765962 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:08:53.766093 1704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.762434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:37917 (local address 127.16.61.3:35745)
0619 11:08:53.762753 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.762769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:53.762866 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:53.763114 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:53.763117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:53.763135 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:53.763248 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.763254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.764012 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.764015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.764891 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:53.764901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.765688 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.765691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:53.765794 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.765820 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 11:08:53.765890 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:08:53.765954 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":46,"thread_start_us":90,"threads_started":1}
I20260619 11:08:53.766799 1498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b37c784b31540ea975c7108a058c892" candidate_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57786d30a9074ec88bee47a0562f4810" is_pre_election: true
I20260619 11:08:53.766799 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b37c784b31540ea975c7108a058c892" candidate_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20916f537d0b4a708caa14a2baf2d1cf" is_pre_election: true
I20260619 11:08:53.767050 1498 raft_consensus.cc:2454] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b86e1cbba44c8f8ec439a79ba51c0b in term 0.
I20260619 11:08:53.767050 1364 raft_consensus.cc:2454] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b86e1cbba44c8f8ec439a79ba51c0b in term 0.
I20260619 11:08:53.767477 1565 leader_election.cc:302] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20916f537d0b4a708caa14a2baf2d1cf, c1b86e1cbba44c8f8ec439a79ba51c0b; no voters:
I20260619 11:08:53.767714 1702 raft_consensus.cc:2781] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:08:53.767824 1702 raft_consensus.cc:482] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:08:53.767886 1702 raft_consensus.cc:3037] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:53.768800 1702 raft_consensus.cc:504] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
I20260619 11:08:53.769120 1702 leader_election.cc:288] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [CANDIDATE]: Term 1 election: Requested vote from peers 20916f537d0b4a708caa14a2baf2d1cf (127.16.61.1:33741), 57786d30a9074ec88bee47a0562f4810 (127.16.61.2:37917)
I20260619 11:08:53.769438 1364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b37c784b31540ea975c7108a058c892" candidate_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20916f537d0b4a708caa14a2baf2d1cf"
I20260619 11:08:53.769547 1498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b37c784b31540ea975c7108a058c892" candidate_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57786d30a9074ec88bee47a0562f4810"
I20260619 11:08:53.769575 1364 raft_consensus.cc:3037] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:53.769698 1498 raft_consensus.cc:3037] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:53.770608 1498 raft_consensus.cc:2454] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b86e1cbba44c8f8ec439a79ba51c0b in term 1.
I20260619 11:08:53.770609 1364 raft_consensus.cc:2454] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b86e1cbba44c8f8ec439a79ba51c0b in term 1.
I20260619 11:08:53.770875 1565 leader_election.cc:302] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20916f537d0b4a708caa14a2baf2d1cf, c1b86e1cbba44c8f8ec439a79ba51c0b; no voters:
I20260619 11:08:53.771070 1702 raft_consensus.cc:2781] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:53.771267 1702 raft_consensus.cc:686] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 1 LEADER]: Becoming Leader. State: Replica: c1b86e1cbba44c8f8ec439a79ba51c0b, State: Running, Role: LEADER
I20260619 11:08:53.771612 1702 consensus_queue.cc:237] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } }
W20260619 11:08:53.774657 1411 tablet.cc:2369] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:08:53.774466 1218 catalog_manager.cc:5495] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b reported cstate change: term changed from 0 to 1, leader changed from <none> to c1b86e1cbba44c8f8ec439a79ba51c0b (127.16.61.3). New cstate: current_term: 1 leader_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b86e1cbba44c8f8ec439a79ba51c0b" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 37425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 } health_report { overall_health: UNKNOWN } } }
I20260619 11:08:53.795872 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.792673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.16.61.3:37425)
0619 11:08:53.792771 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:53.792775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:53.792813 (+ 38us) server_negotiation.cc:407] Connection header received
0619 11:08:53.792960 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:53.792962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:53.793008 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:53.793088 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:08:53.793291 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.793764 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.794535 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:53.794736 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:53.794807 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:53.795106 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:08:53.795457 (+ 351us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:08:53.795544 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:53.795599 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 11:08:53.795650 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 11:08:53.801029 1364 raft_consensus.cc:1261] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 1 FOLLOWER]: Refusing update from remote peer c1b86e1cbba44c8f8ec439a79ba51c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:53.801029 1498 raft_consensus.cc:1261] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 1 FOLLOWER]: Refusing update from remote peer c1b86e1cbba44c8f8ec439a79ba51c0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:53.801674 1702 consensus_queue.cc:1035] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "57786d30a9074ec88bee47a0562f4810" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 37917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:53.801846 1705 consensus_queue.cc:1035] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20916f537d0b4a708caa14a2baf2d1cf" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:53.810467 1709 mvcc.cc:205] Tried to move back new op lower bound from 7298528599242358784 to 7298528599130996736. Current Snapshot: MvccSnapshot[applied={T|T < 7298528599242358784}]
I20260619 11:08:53.812924 1710 mvcc.cc:205] Tried to move back new op lower bound from 7298528599242358784 to 7298528599130996736. Current Snapshot: MvccSnapshot[applied={T|T < 7298528599242358784}]
I20260619 11:08:53.816082 1218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:53.816252 1218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:53.818439 1218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13cf963db2a5419aaa6fcc1f66c509a1: Sending DeleteTablet for 3 replicas of tablet 8b37c784b31540ea975c7108a058c892
I20260619 11:08:53.819121 1344 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b37c784b31540ea975c7108a058c892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:53 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:43146
I20260619 11:08:53.819151 1612 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b37c784b31540ea975c7108a058c892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:53 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:39872
I20260619 11:08:53.819555 1478 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b37c784b31540ea975c7108a058c892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:53 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:51926
I20260619 11:08:53.819693 1718 tablet_replica.cc:330] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf: stopping tablet replica
I20260619 11:08:53.819880 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 1281
I20260619 11:08:53.819980 1718 raft_consensus.cc:2229] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:53.820233 1718 raft_consensus.cc:2258] T 8b37c784b31540ea975c7108a058c892 P 20916f537d0b4a708caa14a2baf2d1cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:53.820463 1719 tablet_replica.cc:330] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: stopping tablet replica
I20260619 11:08:53.820719 1719 raft_consensus.cc:2229] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:08:53.821014 1719 raft_consensus.cc:2258] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:53.822300 1719 ts_tablet_manager.cc:1905] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:53.824764 1720 tablet_replica.cc:330] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810: stopping tablet replica
I20260619 11:08:53.825086 1720 raft_consensus.cc:2229] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:53.825423 1720 raft_consensus.cc:2258] T 8b37c784b31540ea975c7108a058c892 P 57786d30a9074ec88bee47a0562f4810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:53.825848 1719 ts_tablet_manager.cc:1918] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:53.825951 1719 log.cc:1196] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/8b37c784b31540ea975c7108a058c892
I20260619 11:08:53.826326 1719 ts_tablet_manager.cc:1939] T 8b37c784b31540ea975c7108a058c892 P c1b86e1cbba44c8f8ec439a79ba51c0b: Deleting consensus metadata
W20260619 11:08:53.826900 1203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:33741 (error 108)
I20260619 11:08:53.827082 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 1415
I20260619 11:08:53.828615 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.828307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:33741 (local address 127.0.0.1:43152)
0619 11:08:53.828443 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.828521 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:33741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 11:08:53.828804 1203 catalog_manager.cc:4584] TS 20916f537d0b4a708caa14a2baf2d1cf (127.16.61.1:33741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b37c784b31540ea975c7108a058c892: Network error: Client connection negotiation failed: client connection to 127.16.61.1:33741: connect: Connection refused (error 111)
I20260619 11:08:53.829571 1203 catalog_manager.cc:4841] TS c1b86e1cbba44c8f8ec439a79ba51c0b (127.16.61.3:37425): tablet 8b37c784b31540ea975c7108a058c892 (table test-table [id=6ce3ea204a80422c8e0582b7296f003f]) successfully deleted
I20260619 11:08:53.833163 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 1549
I20260619 11:08:53.833292 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:53.833064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:37917 (local address 127.0.0.1:51940)
0619 11:08:53.833159 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:53.833207 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.2:37917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 11:08:53.838124 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 1187
2026-06-19T11:08:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2169 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T11:08:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:08:53Z Disabled control of system clock
I20260619 11:08:53.867597 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:39853
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:39853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:08:53.972966 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:53.973196 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:53.973228 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:53.977087 1725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:08:53.977139 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:53.977155 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:53.977172 1725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:08:53.977186 1725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:08:53.981650 1725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:39853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:39853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:53.982486 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:53.983703 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:53.991708 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:53.991829 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:53.991824 1725 server_base.cc:1034] running on GCE node
W20260619 11:08:53.991708 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:53.992527 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:53.993484 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:53.994637 1725 hybrid_clock.cc:648] HybridClock initialized: now 1781867333994620 us; error 30 us; skew 500 ppm
I20260619 11:08:53.996397 1725 webserver.cc:460] Webserver started at http://127.16.61.62:39303/ using document root <none> and password file <none>
I20260619 11:08:53.996923 1725 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:53.996981 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:53.997129 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:53.998953 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "7c5e6e2c1d834c61968709e16135da7e"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "3e1b92db12fbdbf7fef8868752784345"
server_key_iv: "bbbc69b5e14d22d88944d8bf24169e46"
server_key_version: "encryptionkey@0"
I20260619 11:08:53.999420 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "7c5e6e2c1d834c61968709e16135da7e"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "3e1b92db12fbdbf7fef8868752784345"
server_key_iv: "bbbc69b5e14d22d88944d8bf24169e46"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.002857 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:08:54.005095 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.006224 1725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:08:54.006366 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "7c5e6e2c1d834c61968709e16135da7e"
format_stamp: "Formatted at 2026-06-19 11:08:53 on dist-test-slave-8vcz"
server_key: "3e1b92db12fbdbf7fef8868752784345"
server_key_iv: "bbbc69b5e14d22d88944d8bf24169e46"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.006472 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:54.020627 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:54.021258 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:54.021451 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:54.028106 1725 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:39853
I20260619 11:08:54.028147 1791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:39853 every 8 connection(s)
I20260619 11:08:54.029022 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:08:54.031994 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:54.032437 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 1725
I20260619 11:08:54.032552 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:08:54.032856 16628 external_mini_cluster.cc:1428] Setting key 1431b8f138d1f1ddd4d2acad7852696f
I20260619 11:08:54.038353 1792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e: Bootstrap starting.
I20260619 11:08:54.040876 1792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:54.041728 1792 log.cc:824] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:54.042169 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.034188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.16.61.62:39853)
0619 11:08:54.034792 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.034806 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.034838 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:08:54.035502 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.035518 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.035949 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.036332 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:54.036731 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.037654 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.038539 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.039194 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.039376 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.040187 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:54.040194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:54.040208 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:54.040658 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:54.040729 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:54.040758 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:54.040861 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.040986 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.041266 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260619 11:08:54.043766 1792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e: No bootstrap required, opened a new log
I20260619 11:08:54.047075 1792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e6e2c1d834c61968709e16135da7e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39853 } }
I20260619 11:08:54.047276 1792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:54.047343 1792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5e6e2c1d834c61968709e16135da7e, State: Initialized, Role: FOLLOWER
I20260619 11:08:54.047851 1792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e6e2c1d834c61968709e16135da7e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39853 } }
I20260619 11:08:54.047976 1792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:08:54.048056 1792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:08:54.048158 1792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:54.049156 1792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e6e2c1d834c61968709e16135da7e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39853 } }
I20260619 11:08:54.049494 1792 leader_election.cc:302] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c5e6e2c1d834c61968709e16135da7e; no voters:
I20260619 11:08:54.049800 1792 leader_election.cc:288] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:08:54.049906 1797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:54.050148 1797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [term 1 LEADER]: Becoming Leader. State: Replica: 7c5e6e2c1d834c61968709e16135da7e, State: Running, Role: LEADER
I20260619 11:08:54.050503 1797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e6e2c1d834c61968709e16135da7e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39853 } }
I20260619 11:08:54.050942 1792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:08:54.052192 1798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c5e6e2c1d834c61968709e16135da7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e6e2c1d834c61968709e16135da7e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39853 } } }
I20260619 11:08:54.052232 1799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c5e6e2c1d834c61968709e16135da7e. Latest consensus state: current_term: 1 leader_uuid: "7c5e6e2c1d834c61968709e16135da7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5e6e2c1d834c61968709e16135da7e" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 39853 } } }
I20260619 11:08:54.052412 1798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:54.052436 1799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:54.053356 1807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:08:54.056213 1807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:08:54.061434 1807 catalog_manager.cc:1347] Generated new cluster ID: c7dd749e104e4e6ab794249fb2e9796b
I20260619 11:08:54.061535 1807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:08:54.082073 1807 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:08:54.083178 1807 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:08:54.092272 1807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e: Generated new TSK 0
I20260619 11:08:54.092962 1807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:08:54.100543 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39853
--builtin_ntp_servers=127.16.61.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:08:54.208748 1816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:54.209021 1816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:54.209088 1816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:54.213199 1816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:54.213246 1816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:54.213299 1816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:08:54.217871 1816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:54.218801 1816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:54.220101 1816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:54.229039 1821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:54.229068 1824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:54.229205 1816 server_base.cc:1034] running on GCE node
W20260619 11:08:54.229233 1822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:54.229655 1816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:54.230217 1816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:54.231384 1816 hybrid_clock.cc:648] HybridClock initialized: now 1781867334231372 us; error 23 us; skew 500 ppm
I20260619 11:08:54.233119 1816 webserver.cc:460] Webserver started at http://127.16.61.1:41103/ using document root <none> and password file <none>
I20260619 11:08:54.233664 1816 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:54.233726 1816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:54.233951 1816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:54.235984 1816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "ce30d7a64fca422db9b11a8e356076b6"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "956dbc7ff3edb8041af603b70d3a098b"
server_key_iv: "950e93327bdeaab63d25c2e4a4bb03e3"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.236521 1816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "ce30d7a64fca422db9b11a8e356076b6"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "956dbc7ff3edb8041af603b70d3a098b"
server_key_iv: "950e93327bdeaab63d25c2e4a4bb03e3"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.239960 1816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:08:54.242221 1830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.243360 1816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:54.243494 1816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "ce30d7a64fca422db9b11a8e356076b6"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "956dbc7ff3edb8041af603b70d3a098b"
server_key_iv: "950e93327bdeaab63d25c2e4a4bb03e3"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.243603 1816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:54.262233 1816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:54.262823 1816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:54.263008 1816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:54.263542 1816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:54.264462 1816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:54.264537 1816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.264600 1816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:54.264642 1816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.274124 1816 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:46849
I20260619 11:08:54.274149 1943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:46849 every 8 connection(s)
I20260619 11:08:54.274945 1816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:08:54.276233 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 1816
I20260619 11:08:54.276351 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:08:54.276691 16628 external_mini_cluster.cc:1428] Setting key bf479655d9c7922e30dc299d271023a1
I20260619 11:08:54.278508 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39853
--builtin_ntp_servers=127.16.61.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:08:54.285315 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.276727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:43375 (local address 127.16.61.62:39853)
0619 11:08:54.276876 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.276880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.278083 (+ 1203us) server_negotiation.cc:407] Connection header received
0619 11:08:54.279507 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.279513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.279575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.279693 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:54.280865 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.281392 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.283250 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.283461 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.283533 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.284847 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:54.284850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:54.284852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:54.284864 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:54.284882 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:54.284890 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:54.284943 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.285149 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.285200 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:08:54.286521 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.277221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:39853 (local address 127.16.61.1:43375)
0619 11:08:54.277947 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:54.277995 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:54.279296 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:54.279903 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:54.279912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:54.280025 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:54.280699 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.280723 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.281524 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.281530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.283104 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.283119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.283613 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.283619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.283879 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.284527 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:08:54.284531 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:54.284589 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:08:54.284676 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:54.284975 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:54.284986 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:54.284996 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 11:08:54.285148 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 11:08:54.285460 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":159,"threads_started":1}
I20260619 11:08:54.287894 1944 heartbeater.cc:347] Connected to a master server at 127.16.61.62:39853
I20260619 11:08:54.288136 1944 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:54.288618 1944 heartbeater.cc:511] Master 127.16.61.62:39853 requested a full tablet report, sending...
I20260619 11:08:54.290139 1756 ts_manager.cc:194] Registered new tserver with Master: ce30d7a64fca422db9b11a8e356076b6 (127.16.61.1:46849)
I20260619 11:08:54.291254 1756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.1:43375
W20260619 11:08:54.390959 1947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:54.391201 1947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:54.391265 1947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:54.395231 1947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:54.395277 1947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:54.395359 1947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:08:54.400141 1947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:54.401093 1947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:54.402377 1947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:54.411522 1953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:54.411537 1952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:54.411592 1947 server_base.cc:1034] running on GCE node
W20260619 11:08:54.411537 1955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:54.412104 1947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:54.412729 1947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:54.413964 1947 hybrid_clock.cc:648] HybridClock initialized: now 1781867334413946 us; error 25 us; skew 500 ppm
I20260619 11:08:54.415772 1947 webserver.cc:460] Webserver started at http://127.16.61.2:34715/ using document root <none> and password file <none>
I20260619 11:08:54.416275 1947 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:54.416333 1947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:54.416482 1947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:54.418354 1947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "6db5545861cc4c35a3e5a3de7300a232"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "275be2b49710f11c731a55d3e37aea11"
server_key_iv: "480411b8e82d15ec7d86d77b2d937081"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.418910 1947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "6db5545861cc4c35a3e5a3de7300a232"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "275be2b49710f11c731a55d3e37aea11"
server_key_iv: "480411b8e82d15ec7d86d77b2d937081"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.422386 1947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:08:54.424696 1961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.425971 1947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:54.426101 1947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "6db5545861cc4c35a3e5a3de7300a232"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "275be2b49710f11c731a55d3e37aea11"
server_key_iv: "480411b8e82d15ec7d86d77b2d937081"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.426208 1947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:54.447635 1947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:54.448274 1947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:54.448451 1947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:54.448987 1947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:54.449846 1947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:54.449916 1947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.449985 1947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:54.450027 1947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.459350 1947 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:43233
I20260619 11:08:54.459439 2074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:43233 every 8 connection(s)
I20260619 11:08:54.460260 1947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:08:54.464785 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 1947
I20260619 11:08:54.464911 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:08:54.465185 16628 external_mini_cluster.cc:1428] Setting key 0d71c89ebd3adb3659307ff9c950c03b
I20260619 11:08:54.467262 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39853
--builtin_ntp_servers=127.16.61.20:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:08:54.470201 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.463328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:34541 (local address 127.16.61.62:39853)
0619 11:08:54.463458 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.463462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.463476 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:08:54.464644 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.464648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.464708 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.464810 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:54.465793 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.466515 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.468213 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.468418 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.468498 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.469652 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:54.469656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:54.469659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:54.469674 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:54.469700 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:54.469714 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:54.469796 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.469927 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.469997 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:08:54.470880 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.462500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:39853 (local address 127.16.61.2:34541)
0619 11:08:54.463101 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:54.463163 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:54.464451 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:54.465024 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:54.465039 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:54.465141 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:54.465631 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.465643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.466663 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.466666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.468097 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.468110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.468571 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.468577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.468810 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.469407 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:08:54.469408 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:54.469446 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:08:54.469502 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:54.469800 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:54.469805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:54.469809 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:08:54.469899 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:08:54.470096 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260619 11:08:54.472142 2075 heartbeater.cc:347] Connected to a master server at 127.16.61.62:39853
I20260619 11:08:54.472362 2075 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:54.472821 2075 heartbeater.cc:511] Master 127.16.61.62:39853 requested a full tablet report, sending...
I20260619 11:08:54.473826 1756 ts_manager.cc:194] Registered new tserver with Master: 6db5545861cc4c35a3e5a3de7300a232 (127.16.61.2:43233)
I20260619 11:08:54.474400 1756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.2:34541
W20260619 11:08:54.574536 2078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:54.574798 2078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:54.574865 2078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:54.578845 2078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:54.578922 2078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:54.579010 2078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:08:54.583647 2078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:39853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:54.584618 2078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:54.585896 2078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:54.594911 2084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:54.594983 2083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:54.594920 2086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:54.595105 2078 server_base.cc:1034] running on GCE node
I20260619 11:08:54.595595 2078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:54.596199 2078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:54.597358 2078 hybrid_clock.cc:648] HybridClock initialized: now 1781867334597332 us; error 37 us; skew 500 ppm
I20260619 11:08:54.598975 2078 webserver.cc:460] Webserver started at http://127.16.61.3:35429/ using document root <none> and password file <none>
I20260619 11:08:54.599529 2078 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:54.599619 2078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:54.599867 2078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:54.601737 2078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "c28b0c24c3684e479b5bf016f4ec87f0"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "b7481ee9599a1e1c521a75c882c4cf6f"
server_key_iv: "d0c77622035be9021b90da84f88308a7"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.602264 2078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "c28b0c24c3684e479b5bf016f4ec87f0"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "b7481ee9599a1e1c521a75c882c4cf6f"
server_key_iv: "d0c77622035be9021b90da84f88308a7"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.605625 2078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:08:54.607865 2092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.608971 2078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:54.609117 2078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "c28b0c24c3684e479b5bf016f4ec87f0"
format_stamp: "Formatted at 2026-06-19 11:08:54 on dist-test-slave-8vcz"
server_key: "b7481ee9599a1e1c521a75c882c4cf6f"
server_key_iv: "d0c77622035be9021b90da84f88308a7"
server_key_version: "encryptionkey@0"
I20260619 11:08:54.609223 2078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:54.647100 2078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:54.647776 2078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:54.647976 2078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:54.648537 2078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:54.649417 2078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:54.649489 2078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.649559 2078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:54.649612 2078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:54.658295 2078 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:40181
I20260619 11:08:54.658324 2205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:40181 every 8 connection(s)
I20260619 11:08:54.659190 2078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:08:54.663734 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 2078
I20260619 11:08:54.663826 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:08:54.664103 16628 external_mini_cluster.cc:1428] Setting key 9d6234c373b0343678305fe2a8eee545
I20260619 11:08:54.668249 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.660900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:54019 (local address 127.16.61.62:39853)
0619 11:08:54.661030 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.661033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.661997 (+ 964us) server_negotiation.cc:407] Connection header received
0619 11:08:54.663346 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.663350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.663405 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.663508 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:54.664499 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.665308 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.666674 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.666890 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.666969 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.667896 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:54.667900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:54.667902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:54.667912 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:54.667930 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:54.667937 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:54.667996 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.668103 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.668146 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:08:54.669050 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.661263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:39853 (local address 127.16.61.3:54019)
0619 11:08:54.661860 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:54.661897 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:54.663138 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:54.663692 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:54.663698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:54.663777 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:54.664331 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.664345 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.665448 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.665452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.666566 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.666576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.667006 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.667009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.667137 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.667639 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:08:54.667640 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:54.667710 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:08:54.667767 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:54.668009 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:54.668015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:54.668019 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:08:54.668106 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 11:08:54.668343 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260619 11:08:54.670097 2206 heartbeater.cc:347] Connected to a master server at 127.16.61.62:39853
I20260619 11:08:54.670341 2206 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:54.670810 2206 heartbeater.cc:511] Master 127.16.61.62:39853 requested a full tablet report, sending...
I20260619 11:08:54.671829 1756 ts_manager.cc:194] Registered new tserver with Master: c28b0c24c3684e479b5bf016f4ec87f0 (127.16.61.3:40181)
I20260619 11:08:54.672375 1756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.3:54019
I20260619 11:08:54.677526 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:08:54.682868 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.679376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.16.61.62:39853)
0619 11:08:54.679479 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.679483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.679717 (+ 234us) server_negotiation.cc:407] Connection header received
0619 11:08:54.679847 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.679849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.679885 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.679943 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:54.680275 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.680741 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.681556 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.681762 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.681837 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.682517 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:54.682521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:54.682523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:54.682533 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:54.682548 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:54.682555 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:54.682618 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.682732 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.682775 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:08:54.685523 1756 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:08:54.688393 1756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:08:54.701683 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.696365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.16.61.2:43233)
0619 11:08:54.697347 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.697360 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.697383 (+ 23us) server_negotiation.cc:407] Connection header received
0619 11:08:54.697460 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.697469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.697675 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.697831 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:54.698143 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.699063 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.700528 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.701047 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.701170 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.701372 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.701439 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.701526 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":97,"threads_started":1}
I20260619 11:08:54.701678 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.696242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:43233 (local address 127.0.0.1:55200)
0619 11:08:54.697033 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:54.697080 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:54.697205 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:54.697845 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:54.697853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:54.697881 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:54.698023 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.698029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.699180 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.699183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.700409 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.700418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.701169 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.701172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.701275 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.701309 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:08:54.701404 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 11:08:54.701519 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":47,"thread_start_us":66,"threads_started":1}
I20260619 11:08:54.702176 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.696490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:46849 (local address 127.0.0.1:44900)
0619 11:08:54.697439 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:54.697452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:54.697521 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:54.698207 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:54.698209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:54.698227 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:54.698316 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.698324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.699850 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.699857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.700742 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.700756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.701788 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.701792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.701905 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.701926 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:08:54.701971 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 11:08:54.702031 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":53,"thread_start_us":84,"threads_started":1}
I20260619 11:08:54.702858 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.696861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.16.61.1:46849)
0619 11:08:54.697783 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.697795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.697817 (+ 22us) server_negotiation.cc:407] Connection header received
0619 11:08:54.697887 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.697892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.698059 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.698225 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:54.698454 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.699703 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.700874 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.701644 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.702394 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.702557 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.702597 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.702714 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":116,"threads_started":1}
I20260619 11:08:54.703853 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.696656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40181 (local address 127.0.0.1:55812)
0619 11:08:54.697804 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:54.697817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:54.697892 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:54.698427 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:54.698430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:54.698449 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:54.698529 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.698534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.699880 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.699883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.702073 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.702082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.703472 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.703474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.703536 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.703553 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:08:54.703645 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 11:08:54.703733 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":208,"thread_start_us":94,"threads_started":1}
I20260619 11:08:54.704260 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.697221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.16.61.3:40181)
0619 11:08:54.697655 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.697663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.697979 (+ 316us) server_negotiation.cc:407] Connection header received
0619 11:08:54.698056 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.698064 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.698279 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.698431 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:54.698645 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.699771 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.702192 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.703115 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.703237 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.703418 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.704010 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.704103 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":96,"threads_started":1}
I20260619 11:08:54.704929 2009 tablet_service.cc:1467] Processing CreateTablet for tablet 07dccc2757064306bd68ec7c88ca31b7 (DEFAULT_TABLE table=test-table [id=974532408a0141ccbe549e94633b6a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:54.705860 2009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07dccc2757064306bd68ec7c88ca31b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:54.705463 1878 tablet_service.cc:1467] Processing CreateTablet for tablet 07dccc2757064306bd68ec7c88ca31b7 (DEFAULT_TABLE table=test-table [id=974532408a0141ccbe549e94633b6a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:54.706429 1878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07dccc2757064306bd68ec7c88ca31b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:54.706800 2140 tablet_service.cc:1467] Processing CreateTablet for tablet 07dccc2757064306bd68ec7c88ca31b7 (DEFAULT_TABLE table=test-table [id=974532408a0141ccbe549e94633b6a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:54.707731 2140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07dccc2757064306bd68ec7c88ca31b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:54.712553 2223 tablet_bootstrap.cc:492] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Bootstrap starting.
I20260619 11:08:54.714645 2225 tablet_bootstrap.cc:492] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Bootstrap starting.
I20260619 11:08:54.715130 2223 tablet_bootstrap.cc:654] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:54.715180 2224 tablet_bootstrap.cc:492] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6: Bootstrap starting.
I20260619 11:08:54.715917 2223 log.cc:824] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:54.716719 2225 tablet_bootstrap.cc:654] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:54.717125 2224 tablet_bootstrap.cc:654] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:54.717514 2225 log.cc:824] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:54.717849 2223 tablet_bootstrap.cc:492] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: No bootstrap required, opened a new log
I20260619 11:08:54.717965 2224 log.cc:824] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:54.718111 2223 ts_tablet_manager.cc:1397] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:08:54.719318 2225 tablet_bootstrap.cc:492] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: No bootstrap required, opened a new log
I20260619 11:08:54.719496 2225 ts_tablet_manager.cc:1397] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:08:54.719923 2224 tablet_bootstrap.cc:492] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6: No bootstrap required, opened a new log
I20260619 11:08:54.720100 2224 ts_tablet_manager.cc:1397] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:08:54.721555 2223 raft_consensus.cc:348] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.721791 2223 raft_consensus.cc:374] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:54.721863 2223 raft_consensus.cc:729] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db5545861cc4c35a3e5a3de7300a232, State: Initialized, Role: FOLLOWER
I20260619 11:08:54.722311 2223 consensus_queue.cc:260] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.722918 2075 heartbeater.cc:503] Master 127.16.61.62:39853 was elected leader, sending a full tablet report...
I20260619 11:08:54.723613 2225 raft_consensus.cc:348] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.723944 2225 raft_consensus.cc:374] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:54.724035 2225 raft_consensus.cc:729] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28b0c24c3684e479b5bf016f4ec87f0, State: Initialized, Role: FOLLOWER
I20260619 11:08:54.724164 2223 ts_tablet_manager.cc:1428] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:08:54.724057 2224 raft_consensus.cc:348] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.724347 2224 raft_consensus.cc:374] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:54.724435 2224 raft_consensus.cc:729] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce30d7a64fca422db9b11a8e356076b6, State: Initialized, Role: FOLLOWER
I20260619 11:08:54.724514 2225 consensus_queue.cc:260] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.724814 2224 consensus_queue.cc:260] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.725188 2206 heartbeater.cc:503] Master 127.16.61.62:39853 was elected leader, sending a full tablet report...
I20260619 11:08:54.725432 2225 ts_tablet_manager.cc:1428] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:08:54.725708 2224 ts_tablet_manager.cc:1428] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 11:08:54.725896 1944 heartbeater.cc:503] Master 127.16.61.62:39853 was elected leader, sending a full tablet report...
W20260619 11:08:54.776000 1945 tablet.cc:2369] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 11:08:54.910122 2207 tablet.cc:2369] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:08:54.954114 2231 raft_consensus.cc:482] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:54.954329 2231 raft_consensus.cc:504] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.955506 2231 leader_election.cc:288] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6db5545861cc4c35a3e5a3de7300a232 (127.16.61.2:43233), c28b0c24c3684e479b5bf016f4ec87f0 (127.16.61.3:40181)
I20260619 11:08:54.960732 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.955843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:60219 (local address 127.16.61.2:43233)
0619 11:08:54.955980 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.955985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.956163 (+ 178us) server_negotiation.cc:407] Connection header received
0619 11:08:54.956318 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.956320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.956372 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.956459 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:54.956882 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.957688 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.959471 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.960091 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.960216 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.960261 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.960533 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.960613 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:08:54.960732 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.955905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:34293 (local address 127.16.61.3:40181)
0619 11:08:54.956032 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.956037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.956230 (+ 193us) server_negotiation.cc:407] Connection header received
0619 11:08:54.956317 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.956320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.956372 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.956459 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:54.956882 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.957689 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.959471 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.960098 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.960200 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.960261 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.960533 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.960607 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:08:54.960820 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.955800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:40181 (local address 127.16.61.1:34293)
0619 11:08:54.956143 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:54.956155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:54.956231 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:54.956489 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:54.956492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:54.956534 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:54.956756 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.956765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.957823 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.957827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.959344 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.959349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.960227 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.960231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.960383 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.960420 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 11:08:54.960501 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 11:08:54.960641 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":280,"spinlock_wait_cycles":14592,"thread_start_us":76,"threads_started":1}
I20260619 11:08:54.960820 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.955717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:43233 (local address 127.16.61.1:60219)
0619 11:08:54.956064 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:54.956079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:54.956202 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:54.956489 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:54.956492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:54.956532 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:54.956765 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.956770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.957823 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.957827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.959335 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:54.959348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.960227 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.960231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:54.960379 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.960425 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 11:08:54.960501 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 11:08:54.960641 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":268,"spinlock_wait_cycles":26880,"thread_start_us":100,"threads_started":1}
W20260619 11:08:54.961308 2076 tablet.cc:2369] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:08:54.961570 2160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07dccc2757064306bd68ec7c88ca31b7" candidate_uuid: "ce30d7a64fca422db9b11a8e356076b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" is_pre_election: true
I20260619 11:08:54.961555 2029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07dccc2757064306bd68ec7c88ca31b7" candidate_uuid: "ce30d7a64fca422db9b11a8e356076b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db5545861cc4c35a3e5a3de7300a232" is_pre_election: true
I20260619 11:08:54.961843 2029 raft_consensus.cc:2454] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce30d7a64fca422db9b11a8e356076b6 in term 0.
I20260619 11:08:54.961856 2160 raft_consensus.cc:2454] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce30d7a64fca422db9b11a8e356076b6 in term 0.
I20260619 11:08:54.962319 1833 leader_election.cc:302] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6db5545861cc4c35a3e5a3de7300a232, ce30d7a64fca422db9b11a8e356076b6; no voters:
I20260619 11:08:54.962586 2231 raft_consensus.cc:2781] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:08:54.962680 2231 raft_consensus.cc:482] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:08:54.962739 2231 raft_consensus.cc:3037] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:54.963737 2231 raft_consensus.cc:504] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.964066 2231 leader_election.cc:288] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [CANDIDATE]: Term 1 election: Requested vote from peers 6db5545861cc4c35a3e5a3de7300a232 (127.16.61.2:43233), c28b0c24c3684e479b5bf016f4ec87f0 (127.16.61.3:40181)
I20260619 11:08:54.964494 2029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07dccc2757064306bd68ec7c88ca31b7" candidate_uuid: "ce30d7a64fca422db9b11a8e356076b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db5545861cc4c35a3e5a3de7300a232"
I20260619 11:08:54.964602 2029 raft_consensus.cc:3037] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:54.965519 2029 raft_consensus.cc:2454] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce30d7a64fca422db9b11a8e356076b6 in term 1.
I20260619 11:08:54.965788 2160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07dccc2757064306bd68ec7c88ca31b7" candidate_uuid: "ce30d7a64fca422db9b11a8e356076b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c28b0c24c3684e479b5bf016f4ec87f0"
I20260619 11:08:54.965822 1833 leader_election.cc:302] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6db5545861cc4c35a3e5a3de7300a232, ce30d7a64fca422db9b11a8e356076b6; no voters:
I20260619 11:08:54.965902 2160 raft_consensus.cc:3037] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:54.965981 2231 raft_consensus.cc:2781] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:54.966171 2231 raft_consensus.cc:686] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 1 LEADER]: Becoming Leader. State: Replica: ce30d7a64fca422db9b11a8e356076b6, State: Running, Role: LEADER
I20260619 11:08:54.966503 2231 consensus_queue.cc:237] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } }
I20260619 11:08:54.966965 2160 raft_consensus.cc:2454] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce30d7a64fca422db9b11a8e356076b6 in term 1.
I20260619 11:08:54.969395 1754 catalog_manager.cc:5495] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce30d7a64fca422db9b11a8e356076b6 (127.16.61.1). New cstate: current_term: 1 leader_uuid: "ce30d7a64fca422db9b11a8e356076b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce30d7a64fca422db9b11a8e356076b6" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 46849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 } health_report { overall_health: UNKNOWN } } }
I20260619 11:08:54.991870 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:54.988307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.16.61.1:46849)
0619 11:08:54.988435 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:54.988441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:54.988455 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:08:54.988654 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:54.988656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:54.988708 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:54.988814 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:08:54.989048 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.989528 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.990364 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:54.990573 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:54.990661 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:54.990955 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:08:54.991340 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:08:54.991428 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:54.991564 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 11:08:54.991693 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:08:54.997502 2160 raft_consensus.cc:1261] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 1 FOLLOWER]: Refusing update from remote peer ce30d7a64fca422db9b11a8e356076b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:54.997503 2029 raft_consensus.cc:1261] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 1 FOLLOWER]: Refusing update from remote peer ce30d7a64fca422db9b11a8e356076b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:54.998314 2234 consensus_queue.cc:1035] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c28b0c24c3684e479b5bf016f4ec87f0" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:54.998499 2231 consensus_queue.cc:1035] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6db5545861cc4c35a3e5a3de7300a232" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 43233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:55.005792 2238 mvcc.cc:205] Tried to move back new op lower bound from 7298528604142649344 to 7298528604025548800. Current Snapshot: MvccSnapshot[applied={T|T < 7298528604142649344}]
I20260619 11:08:55.008279 2239 mvcc.cc:205] Tried to move back new op lower bound from 7298528604142649344 to 7298528604025548800. Current Snapshot: MvccSnapshot[applied={T|T < 7298528604142649344}]
I20260619 11:08:55.013918 1754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:55.014052 1754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:55.016423 1754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c5e6e2c1d834c61968709e16135da7e: Sending DeleteTablet for 3 replicas of tablet 07dccc2757064306bd68ec7c88ca31b7
I20260619 11:08:55.017201 1878 tablet_service.cc:1514] Processing DeleteTablet for tablet 07dccc2757064306bd68ec7c88ca31b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:55 UTC) from {username='slave'} at 127.0.0.1:44900
I20260619 11:08:55.017297 2140 tablet_service.cc:1514] Processing DeleteTablet for tablet 07dccc2757064306bd68ec7c88ca31b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:55 UTC) from {username='slave'} at 127.0.0.1:55812
I20260619 11:08:55.017616 2246 tablet_replica.cc:330] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6: stopping tablet replica
I20260619 11:08:55.017732 2009 tablet_service.cc:1514] Processing DeleteTablet for tablet 07dccc2757064306bd68ec7c88ca31b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:55 UTC) from {username='slave'} at 127.0.0.1:55200
I20260619 11:08:55.017887 2246 raft_consensus.cc:2229] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:08:55.018198 2246 raft_consensus.cc:2258] T 07dccc2757064306bd68ec7c88ca31b7 P ce30d7a64fca422db9b11a8e356076b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:55.018287 2248 tablet_replica.cc:330] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: stopping tablet replica
I20260619 11:08:55.018350 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 1816
I20260619 11:08:55.018520 2247 tablet_replica.cc:330] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: stopping tablet replica
I20260619 11:08:55.019008 2248 raft_consensus.cc:2229] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:55.019313 2248 raft_consensus.cc:2258] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:55.019528 2247 raft_consensus.cc:2229] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:55.019832 2247 raft_consensus.cc:2258] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:55.021127 2248 ts_tablet_manager.cc:1905] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:55.021158 2247 ts_tablet_manager.cc:1905] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:55.023258 2248 ts_tablet_manager.cc:1918] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:55.023367 2248 log.cc:1196] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/07dccc2757064306bd68ec7c88ca31b7
I20260619 11:08:55.023733 2248 ts_tablet_manager.cc:1939] T 07dccc2757064306bd68ec7c88ca31b7 P 6db5545861cc4c35a3e5a3de7300a232: Deleting consensus metadata
I20260619 11:08:55.024461 2247 ts_tablet_manager.cc:1918] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:55.024556 2247 log.cc:1196] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/07dccc2757064306bd68ec7c88ca31b7
I20260619 11:08:55.024889 2247 ts_tablet_manager.cc:1939] T 07dccc2757064306bd68ec7c88ca31b7 P c28b0c24c3684e479b5bf016f4ec87f0: Deleting consensus metadata
I20260619 11:08:55.024907 1742 catalog_manager.cc:4841] TS 6db5545861cc4c35a3e5a3de7300a232 (127.16.61.2:43233): tablet 07dccc2757064306bd68ec7c88ca31b7 (table test-table [id=974532408a0141ccbe549e94633b6a04]) successfully deleted
I20260619 11:08:55.026022 1743 catalog_manager.cc:4841] TS c28b0c24c3684e479b5bf016f4ec87f0 (127.16.61.3:40181): tablet 07dccc2757064306bd68ec7c88ca31b7 (table test-table [id=974532408a0141ccbe549e94633b6a04]) successfully deleted
W20260619 11:08:55.026141 1741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:46849 (error 108)
I20260619 11:08:55.026419 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 1947
I20260619 11:08:55.028038 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:55.027732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:46849 (local address 127.0.0.1:44922)
0619 11:08:55.027860 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:55.027941 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 11:08:55.028214 1741 catalog_manager.cc:4584] TS ce30d7a64fca422db9b11a8e356076b6 (127.16.61.1:46849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07dccc2757064306bd68ec7c88ca31b7: Network error: Client connection negotiation failed: client connection to 127.16.61.1:46849: connect: Connection refused (error 111)
I20260619 11:08:55.031962 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 2078
I20260619 11:08:55.037698 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 1725
2026-06-19T11:08:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1198 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:08:55 dist-test-slave-8vcz krb5kdc[2252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:08:55 dist-test-slave-8vcz krb5kdc[2252](info): set up 2 sockets
Jun 19 11:08:55 dist-test-slave-8vcz krb5kdc[2252](info): commencing operation
krb5kdc: starting...
W20260619 11:08:56.063416 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:08:56 dist-test-slave-8vcz krb5kdc[2252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:08:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:08:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:08:56.179844 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41385
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:56.285215 2265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:56.285456 2265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:56.285495 2265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:56.289414 2265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:08:56.289463 2265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:56.289482 2265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:56.289498 2265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:08:56.289554 2265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:08:56.294728 2265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:56.295742 2265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:56.296693 2265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:56.305051 2270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:56.305030 2273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:56.305030 2271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:56.305153 2265 server_base.cc:1034] running on GCE node
I20260619 11:08:56.305816 2265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:56.306774 2265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:56.307928 2265 hybrid_clock.cc:648] HybridClock initialized: now 1781867336307910 us; error 29 us; skew 500 ppm
Jun 19 11:08:56 dist-test-slave-8vcz krb5kdc[2252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:56.310585 2265 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:08:56.311414 2265 webserver.cc:460] Webserver started at http://127.16.61.62:41061/ using document root <none> and password file <none>
I20260619 11:08:56.312011 2265 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:56.312100 2265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:56.312323 2265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:56.314219 2265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "c129586e4f3f43a0ae0c641c7dfa09f4"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "9261f955ee57b55db14ecb06894152d0"
server_key_iv: "65791540ce548a4615f4ab9b4430f080"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.314790 2265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "c129586e4f3f43a0ae0c641c7dfa09f4"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "9261f955ee57b55db14ecb06894152d0"
server_key_iv: "65791540ce548a4615f4ab9b4430f080"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.318202 2265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:08:56.320429 2280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:56.321517 2265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 11:08:56.321653 2265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "c129586e4f3f43a0ae0c641c7dfa09f4"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "9261f955ee57b55db14ecb06894152d0"
server_key_iv: "65791540ce548a4615f4ab9b4430f080"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.321770 2265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:56.332556 2265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:56.335541 2265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:56.335789 2265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:56.342453 2265 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:41385
I20260619 11:08:56.342473 2332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:41385 every 8 connection(s)
I20260619 11:08:56.343336 2265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:08:56.345325 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 2265
I20260619 11:08:56.345470 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:08:56.345803 16628 external_mini_cluster.cc:1428] Setting key b84bd37fc47d9f779b64e12ca36b78fa
I20260619 11:08:56.346238 2333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:56.352048 2333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4: Bootstrap starting.
Jun 19 11:08:56 dist-test-slave-8vcz krb5kdc[2252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:56.354295 2333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:56.355086 2333 log.cc:824] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:56.357132 2333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4: No bootstrap required, opened a new log
I20260619 11:08:56.359473 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:56.347174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.16.61.62:41385)
0619 11:08:56.347604 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:56.347618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:56.347652 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:08:56.348346 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:56.348362 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:56.348660 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:56.348971 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:56.349750 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.350701 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.351554 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.352080 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.352278 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:56.354878 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:56.354883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:56.354895 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:56.354920 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:56.356757 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:56.357264 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:56.357269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:56.357273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:56.357376 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:56.357687 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:56.357689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:56.357690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:56.358102 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:56.358236 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:56.358525 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 11:08:56.358805 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":144,"threads_started":1}
I20260619 11:08:56.360899 2333 raft_consensus.cc:348] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129586e4f3f43a0ae0c641c7dfa09f4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41385 } }
I20260619 11:08:56.361146 2333 raft_consensus.cc:374] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:56.361235 2333 raft_consensus.cc:729] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c129586e4f3f43a0ae0c641c7dfa09f4, State: Initialized, Role: FOLLOWER
I20260619 11:08:56.361694 2333 consensus_queue.cc:260] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129586e4f3f43a0ae0c641c7dfa09f4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41385 } }
I20260619 11:08:56.361845 2333 raft_consensus.cc:388] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:08:56.361913 2333 raft_consensus.cc:482] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:08:56.362010 2333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:56.362871 2333 raft_consensus.cc:504] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129586e4f3f43a0ae0c641c7dfa09f4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41385 } }
I20260619 11:08:56.363202 2333 leader_election.cc:302] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c129586e4f3f43a0ae0c641c7dfa09f4; no voters:
I20260619 11:08:56.363493 2333 leader_election.cc:288] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:08:56.363608 2338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:56.363910 2338 raft_consensus.cc:686] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [term 1 LEADER]: Becoming Leader. State: Replica: c129586e4f3f43a0ae0c641c7dfa09f4, State: Running, Role: LEADER
I20260619 11:08:56.364248 2338 consensus_queue.cc:237] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129586e4f3f43a0ae0c641c7dfa09f4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41385 } }
I20260619 11:08:56.364686 2333 sys_catalog.cc:564] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:08:56.365777 2340 sys_catalog.cc:455] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c129586e4f3f43a0ae0c641c7dfa09f4. Latest consensus state: current_term: 1 leader_uuid: "c129586e4f3f43a0ae0c641c7dfa09f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129586e4f3f43a0ae0c641c7dfa09f4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41385 } } }
I20260619 11:08:56.365891 2340 sys_catalog.cc:458] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:56.366227 2339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c129586e4f3f43a0ae0c641c7dfa09f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c129586e4f3f43a0ae0c641c7dfa09f4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 41385 } } }
I20260619 11:08:56.366312 2339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4 [sys.catalog]: This master's current role is: LEADER
W20260619 11:08:56.369168 2353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:08:56.369230 2353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:08:56.369313 2347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:08:56.369990 2347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:08:56.375763 2347 catalog_manager.cc:1347] Generated new cluster ID: 61d6c1c19b7541f8bfe1f3c95f8676fe
I20260619 11:08:56.375830 2347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:08:56.387856 2347 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:08:56.388691 2347 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:08:56.393402 2347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4: Generated new TSK 0
I20260619 11:08:56.393980 2347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:08:56.426043 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41385
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:56.540752 2359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:56.540999 2359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:56.541069 2359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:56.545019 2359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:56.545091 2359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:56.545176 2359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:08:56.550132 2359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:56.551059 2359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:56.551978 2359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:56.561523 2367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:56.561527 2364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:56.561761 2359 server_base.cc:1034] running on GCE node
W20260619 11:08:56.561772 2365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:56.562175 2359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:56.562763 2359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:56.563958 2359 hybrid_clock.cc:648] HybridClock initialized: now 1781867336563946 us; error 23 us; skew 500 ppm
Jun 19 11:08:56 dist-test-slave-8vcz krb5kdc[2252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:56.566761 2359 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:08:56.567492 2359 webserver.cc:460] Webserver started at http://127.16.61.1:46031/ using document root <none> and password file <none>
I20260619 11:08:56.568024 2359 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:56.568085 2359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:56.568243 2359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:56.570040 2359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "e5f963320f7c4268a290051efddd61fb"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "d8ae32f68817000c5027cacdd317c34c"
server_key_iv: "52742e551186234cc662beee56619d52"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.570526 2359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "e5f963320f7c4268a290051efddd61fb"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "d8ae32f68817000c5027cacdd317c34c"
server_key_iv: "52742e551186234cc662beee56619d52"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.573954 2359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:08:56.576211 2374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:56.577483 2359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:56.577587 2359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "e5f963320f7c4268a290051efddd61fb"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "d8ae32f68817000c5027cacdd317c34c"
server_key_iv: "52742e551186234cc662beee56619d52"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.577673 2359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:56.593463 2359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:56.596663 2359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:56.596855 2359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:56.597383 2359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:56.598129 2359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:56.598201 2359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:56.598276 2359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:56.598330 2359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:56.607252 2359 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:34803
I20260619 11:08:56.607314 2487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:34803 every 8 connection(s)
I20260619 11:08:56.608134 2359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:08:56.612471 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 2359
I20260619 11:08:56.612614 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:08:56.612898 16628 external_mini_cluster.cc:1428] Setting key f28418dca23d2a267a0de0e7f93de966
Jun 19 11:08:56 dist-test-slave-8vcz krb5kdc[2252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:56.621546 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:56.609785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:53965 (local address 127.16.61.62:41385)
0619 11:08:56.609943 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:56.609948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:56.610684 (+ 736us) server_negotiation.cc:407] Connection header received
0619 11:08:56.611973 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:56.611975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:56.612021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:56.612092 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:56.613223 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.613870 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.615140 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.615381 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.615493 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:56.618081 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:56.618087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:56.618095 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:56.618119 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:56.619519 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:56.620054 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:56.620058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:56.620060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:56.620143 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:56.620480 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:56.620483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:56.620485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:56.620665 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:56.620771 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:56.621257 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 11:08:56.621380 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
I20260619 11:08:56.622323 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:56.610127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41385 (local address 127.16.61.1:53965)
0619 11:08:56.610531 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:56.610570 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:56.611798 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:56.612218 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:56.612224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:56.612536 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:56.613080 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:56.613095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.614016 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.614020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:56.614932 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:56.614942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.615525 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.615529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:56.615756 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:56.616288 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:56.616303 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:56.617918 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:56.619638 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:56.619646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:56.619696 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:56.619950 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:56.620236 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:56.620240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:56.620242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:56.620379 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:56.620768 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:56.620774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:56.621021 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 11:08:56.621201 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 11:08:56.621497 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
I20260619 11:08:56.623507 2488 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41385
I20260619 11:08:56.623806 2488 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:56.624261 2488 heartbeater.cc:511] Master 127.16.61.62:41385 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
I20260619 11:08:56.625864 2297 ts_manager.cc:194] Registered new tserver with Master: e5f963320f7c4268a290051efddd61fb (127.16.61.1:34803)
I20260619 11:08:56.626982 2297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:53965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:08:56.638780 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41385
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:56.743410 2493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:56.743669 2493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:56.743804 2493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:56.747750 2493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:56.747820 2493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:56.747915 2493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:08:56.753144 2493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:56.754072 2493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:56.754997 2493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:56.763849 2498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:56.763918 2493 server_base.cc:1034] running on GCE node
W20260619 11:08:56.763876 2499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:56.763849 2501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:56.764465 2493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:56.764990 2493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:56.766139 2493 hybrid_clock.cc:648] HybridClock initialized: now 1781867336766105 us; error 50 us; skew 500 ppm
Jun 19 11:08:56 dist-test-slave-8vcz krb5kdc[2252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:56.772441 2493 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:08:56.773300 2493 webserver.cc:460] Webserver started at http://127.16.61.2:35621/ using document root <none> and password file <none>
I20260619 11:08:56.773785 2493 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:56.773839 2493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:56.773991 2493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:56.775908 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "60bffce0294c4234b3ac7b942ac8c1a4"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "ba04fcefd87065c9344b6e5a5c0007b8"
server_key_iv: "f08d376f3fe32edd2da971064c30d2ea"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.776391 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "60bffce0294c4234b3ac7b942ac8c1a4"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "ba04fcefd87065c9344b6e5a5c0007b8"
server_key_iv: "f08d376f3fe32edd2da971064c30d2ea"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.779654 2493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:08:56.781947 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:56.783098 2493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:08:56.783313 2493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "60bffce0294c4234b3ac7b942ac8c1a4"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "ba04fcefd87065c9344b6e5a5c0007b8"
server_key_iv: "f08d376f3fe32edd2da971064c30d2ea"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.783412 2493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:56.805413 2493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:56.808473 2493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:56.808662 2493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:56.809201 2493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:56.809957 2493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:56.810000 2493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:56.810061 2493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:56.810096 2493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:56.818826 2493 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:39891
I20260619 11:08:56.818849 2621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:39891 every 8 connection(s)
I20260619 11:08:56.819608 2493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:08:56.824046 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 2493
I20260619 11:08:56.824183 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:08:56.824499 16628 external_mini_cluster.cc:1428] Setting key 902ed6c5f25a4fe31e614470762a2d92
Jun 19 11:08:56 dist-test-slave-8vcz krb5kdc[2252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
I20260619 11:08:56.833186 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:56.821327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:51367 (local address 127.16.61.62:41385)
0619 11:08:56.821449 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:56.821454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:56.822216 (+ 762us) server_negotiation.cc:407] Connection header received
0619 11:08:56.823479 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:56.823482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:56.823538 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:56.823617 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:56.825126 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.825840 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.826999 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.827249 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.827358 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:56.829770 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:56.829773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:56.829775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:56.829797 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:56.831207 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:56.831737 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:56.831748 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:56.831750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:56.831831 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:56.832165 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:56.832168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:56.832170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:56.832348 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:56.832455 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:56.832909 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 11:08:56.833026 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:08:56.833869 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:56.821683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41385 (local address 127.16.61.2:51367)
0619 11:08:56.822081 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:56.822119 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:56.823285 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:56.823822 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:56.823833 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:56.824308 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:56.824980 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:56.825006 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.825964 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.825970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:56.826855 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:56.826865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:56.827392 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:56.827396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:56.827557 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:56.828171 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:56.828191 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:56.829597 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:56.831340 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:56.831346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:56.831358 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:56.831604 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:56.831924 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:56.831929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:56.831931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:56.832064 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:56.832450 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:56.832456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:56.832681 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 11:08:56.832851 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 11:08:56.833135 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260619 11:08:56.834985 2622 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41385
I20260619 11:08:56.835177 2622 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
I20260619 11:08:56.835716 2622 heartbeater.cc:511] Master 127.16.61.62:41385 requested a full tablet report, sending...
I20260619 11:08:56.836716 2297 ts_manager.cc:194] Registered new tserver with Master: 60bffce0294c4234b3ac7b942ac8c1a4 (127.16.61.2:39891)
I20260619 11:08:56.837324 2297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:51367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:08:56.850457 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41385
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:56.954698 2627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:56.954923 2627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:56.954964 2627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:56.959004 2627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:56.959053 2627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:56.959158 2627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:08:56.964250 2627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:56.965207 2627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:56.966226 2627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:56.975332 2635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:56.975469 2632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:56.975379 2627 server_base.cc:1034] running on GCE node
W20260619 11:08:56.975579 2633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:56.975975 2627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:56.976588 2627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:56.977770 2627 hybrid_clock.cc:648] HybridClock initialized: now 1781867336977741 us; error 44 us; skew 500 ppm
Jun 19 11:08:56 dist-test-slave-8vcz krb5kdc[2252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:56.980341 2627 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:08:56.981235 2627 webserver.cc:460] Webserver started at http://127.16.61.3:46161/ using document root <none> and password file <none>
I20260619 11:08:56.981823 2627 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:56.981923 2627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:56.982177 2627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:56.984141 2627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "01f62445c2d946b3a7c3f8d7519a8bea"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "a1a28fe1957f38d1e0197c2f99faa5fd"
server_key_iv: "20140f4773934fba84b6bce7857306b2"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.984680 2627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "01f62445c2d946b3a7c3f8d7519a8bea"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "a1a28fe1957f38d1e0197c2f99faa5fd"
server_key_iv: "20140f4773934fba84b6bce7857306b2"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.988112 2627 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:08:56.990638 2642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:56.991791 2627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:56.991904 2627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "01f62445c2d946b3a7c3f8d7519a8bea"
format_stamp: "Formatted at 2026-06-19 11:08:56 on dist-test-slave-8vcz"
server_key: "a1a28fe1957f38d1e0197c2f99faa5fd"
server_key_iv: "20140f4773934fba84b6bce7857306b2"
server_key_version: "encryptionkey@0"
I20260619 11:08:56.992015 2627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:57.001637 2627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:57.004307 2627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:57.004493 2627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:57.005012 2627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:57.005759 2627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:57.005828 2627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.005892 2627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:57.005944 2627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.014420 2627 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:33009
I20260619 11:08:57.014444 2755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:33009 every 8 connection(s)
I20260619 11:08:57.015213 2627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:08:57.015652 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 2627
I20260619 11:08:57.015784 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:08:57.016098 16628 external_mini_cluster.cc:1428] Setting key 8b88a5cbbf5512fbca335605b3d08fd7
Jun 19 11:08:57 dist-test-slave-8vcz krb5kdc[2252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867336, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:57.028157 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.017031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:59745 (local address 127.16.61.62:41385)
0619 11:08:57.017176 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.017180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.018103 (+ 923us) server_negotiation.cc:407] Connection header received
0619 11:08:57.019428 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.019430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.019474 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.019539 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:57.020657 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.021173 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.022206 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.022409 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.022499 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.024890 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:57.024893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:57.024895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:57.024916 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:57.026456 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:57.026891 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:57.026894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:57.026895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:57.026966 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:57.027236 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:57.027237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:57.027239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:57.027396 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:57.027467 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.027934 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.028028 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:08:57.029014 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.017494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:41385 (local address 127.16.61.3:59745)
0619 11:08:57.017969 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.018017 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:57.019245 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:57.019752 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:57.019761 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:57.020097 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:57.020533 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.020546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.021274 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.021276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.022107 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.022116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.022524 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.022526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.022653 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.023216 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:57.023231 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:57.024751 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:57.026552 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:57.026556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:57.026566 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:57.026811 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:57.027041 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:57.027044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:57.027045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:57.027160 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:57.027478 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:57.027483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:57.027725 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 11:08:57.027888 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 11:08:57.028150 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260619 11:08:57.030078 2756 heartbeater.cc:347] Connected to a master server at 127.16.61.62:41385
I20260619 11:08:57.030311 2756 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:57.030746 2756 heartbeater.cc:511] Master 127.16.61.62:41385 requested a full tablet report, sending...
I20260619 11:08:57.031797 2296 ts_manager.cc:194] Registered new tserver with Master: 01f62445c2d946b3a7c3f8d7519a8bea (127.16.61.3:33009)
I20260619 11:08:57.032368 2296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:59745
I20260619 11:08:57.041582 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:08:57.050438 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.043402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.16.61.62:41385)
0619 11:08:57.043534 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.043538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.043660 (+ 122us) server_negotiation.cc:407] Connection header received
0619 11:08:57.043807 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.043809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.043845 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.043908 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:57.044514 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.045010 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.045781 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.045980 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.046050 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.047396 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:57.047399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:57.047402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:57.047423 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:57.048759 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:57.049288 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:57.049291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:57.049292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:57.049360 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:57.049635 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:57.049638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:57.049639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:57.049812 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:57.049924 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.050153 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.050259 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:08:57.053258 2296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:08:57.056197 2296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:08:57.070616 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.064451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:34803 (local address 127.0.0.1:50406)
0619 11:08:57.064740 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.064753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:57.064834 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:57.065690 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:57.065693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:57.065710 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:57.065816 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.065822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.067360 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.067363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.068924 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.068935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.069906 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.069910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.070112 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.070171 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 11:08:57.070281 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:08:57.070446 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":310,"thread_start_us":90,"threads_started":1}
I20260619 11:08:57.070679 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.064610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.16.61.1:34803)
0619 11:08:57.065172 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.065182 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.065200 (+ 18us) server_negotiation.cc:407] Connection header received
0619 11:08:57.065258 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.065262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.065402 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.065523 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:57.065951 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.067129 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.069044 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.069793 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.069893 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.070020 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.070331 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.070533 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":124,"threads_started":1}
I20260619 11:08:57.070616 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.063753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:33009 (local address 127.0.0.1:40688)
0619 11:08:57.064061 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.064102 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:57.064213 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:57.066217 (+ 2004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:57.066220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:57.066238 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:57.066341 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.066348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.067317 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.067320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.068784 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.068793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.069906 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.069910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.070109 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.070171 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 11:08:57.070281 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 11:08:57.070449 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":387,"thread_start_us":88,"threads_started":1}
I20260619 11:08:57.071399 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.064579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.16.61.3:33009)
0619 11:08:57.065726 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.065739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.065759 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:08:57.065831 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.065835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.066006 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.066244 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:57.066464 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.067180 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.068911 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.069760 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.069888 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.070020 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.070850 (+ 830us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.071219 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":102,"threads_started":1}
I20260619 11:08:57.071597 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.063849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:39891 (local address 127.0.0.1:57482)
0619 11:08:57.064134 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.064149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:57.064234 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:57.065492 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:57.065499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:57.065525 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:57.065662 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.065670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.067273 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.067279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.068924 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.068935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.071165 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.071169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.071255 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.071277 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:08:57.071349 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 11:08:57.071420 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":128,"thread_start_us":101,"threads_started":1}
I20260619 11:08:57.071919 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.064570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.16.61.2:39891)
0619 11:08:57.065015 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.065041 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.065067 (+ 26us) server_negotiation.cc:407] Connection header received
0619 11:08:57.065144 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.065150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.065345 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.065505 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:57.065798 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.067129 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.070086 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.070850 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.070952 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.071099 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.071708 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.071811 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":84,"threads_started":1}
I20260619 11:08:57.073899 2422 tablet_service.cc:1467] Processing CreateTablet for tablet a1c27e5d4e5146419a542b958b087854 (DEFAULT_TABLE table=test-table [id=410e26edf7144e6585a22100586fc8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:57.074357 2556 tablet_service.cc:1467] Processing CreateTablet for tablet a1c27e5d4e5146419a542b958b087854 (DEFAULT_TABLE table=test-table [id=410e26edf7144e6585a22100586fc8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:57.074378 2690 tablet_service.cc:1467] Processing CreateTablet for tablet a1c27e5d4e5146419a542b958b087854 (DEFAULT_TABLE table=test-table [id=410e26edf7144e6585a22100586fc8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:57.075003 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c27e5d4e5146419a542b958b087854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:57.075235 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c27e5d4e5146419a542b958b087854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:57.075290 2690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c27e5d4e5146419a542b958b087854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:57.080956 2773 tablet_bootstrap.cc:492] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: Bootstrap starting.
I20260619 11:08:57.082231 2774 tablet_bootstrap.cc:492] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb: Bootstrap starting.
I20260619 11:08:57.083642 2773 tablet_bootstrap.cc:654] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:57.083993 2775 tablet_bootstrap.cc:492] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: Bootstrap starting.
I20260619 11:08:57.084367 2774 tablet_bootstrap.cc:654] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:57.084508 2773 log.cc:824] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:57.085594 2774 log.cc:824] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:57.085917 2775 tablet_bootstrap.cc:654] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:57.086644 2773 tablet_bootstrap.cc:492] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: No bootstrap required, opened a new log
I20260619 11:08:57.086843 2773 ts_tablet_manager.cc:1397] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 11:08:57.086937 2775 log.cc:824] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:57.088588 2775 tablet_bootstrap.cc:492] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: No bootstrap required, opened a new log
I20260619 11:08:57.088764 2775 ts_tablet_manager.cc:1397] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:08:57.089977 2773 raft_consensus.cc:348] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.090210 2773 raft_consensus.cc:374] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:57.090267 2773 raft_consensus.cc:729] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60bffce0294c4234b3ac7b942ac8c1a4, State: Initialized, Role: FOLLOWER
I20260619 11:08:57.090690 2773 consensus_queue.cc:260] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.091332 2622 heartbeater.cc:503] Master 127.16.61.62:41385 was elected leader, sending a full tablet report...
I20260619 11:08:57.091631 2773 ts_tablet_manager.cc:1428] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 11:08:57.091698 2775 raft_consensus.cc:348] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.092010 2775 raft_consensus.cc:374] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:57.092111 2775 raft_consensus.cc:729] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f62445c2d946b3a7c3f8d7519a8bea, State: Initialized, Role: FOLLOWER
I20260619 11:08:57.092585 2775 consensus_queue.cc:260] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.093243 2756 heartbeater.cc:503] Master 127.16.61.62:41385 was elected leader, sending a full tablet report...
I20260619 11:08:57.093263 2774 tablet_bootstrap.cc:492] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb: No bootstrap required, opened a new log
I20260619 11:08:57.093444 2774 ts_tablet_manager.cc:1397] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260619 11:08:57.093623 2775 ts_tablet_manager.cc:1428] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 11:08:57.097025 2774 raft_consensus.cc:348] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.097200 2774 raft_consensus.cc:374] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:57.097280 2774 raft_consensus.cc:729] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f963320f7c4268a290051efddd61fb, State: Initialized, Role: FOLLOWER
I20260619 11:08:57.097608 2774 consensus_queue.cc:260] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.098174 2488 heartbeater.cc:503] Master 127.16.61.62:41385 was elected leader, sending a full tablet report...
I20260619 11:08:57.098421 2774 ts_tablet_manager.cc:1428] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260619 11:08:57.109189 2489 tablet.cc:2369] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:08:57.140657 2780 raft_consensus.cc:482] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:57.140872 2780 raft_consensus.cc:504] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.142019 2780 leader_election.cc:288] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60bffce0294c4234b3ac7b942ac8c1a4 (127.16.61.2:39891), e5f963320f7c4268a290051efddd61fb (127.16.61.1:34803)
I20260619 11:08:57.145714 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.142519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:56909 (local address 127.16.61.2:39891)
0619 11:08:57.142638 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.142642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.142658 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:08:57.142707 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.142710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.142769 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.142835 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:57.143146 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.143638 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.144762 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.145165 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.145292 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.145353 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.145527 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.145597 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:08:57.145715 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.142206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:39891 (local address 127.16.61.3:56909)
0619 11:08:57.142376 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.142396 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:57.142491 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:57.142860 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:57.142862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:57.142876 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:57.143005 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.143012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.143796 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.143801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.144631 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.144639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.145292 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.145296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.145398 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.145426 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:08:57.145513 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 11:08:57.145581 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":17}
I20260619 11:08:57.146322 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.142519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:45771 (local address 127.16.61.1:34803)
0619 11:08:57.142637 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.142642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.143235 (+ 593us) server_negotiation.cc:407] Connection header received
0619 11:08:57.143352 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.143355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.143398 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.143488 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:57.143773 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.144260 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.145317 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.145705 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.145826 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.145895 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.146138 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.146211 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 11:08:57.146318 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.142206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:34803 (local address 127.16.61.3:45771)
0619 11:08:57.143148 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.143160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:57.143265 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:57.143499 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:57.143502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:57.143516 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:57.143635 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.143640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.144379 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.144382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.145195 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.145203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.145911 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.145918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.146025 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.146050 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 11:08:57.146111 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 11:08:57.146182 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":108,"thread_start_us":124,"threads_started":1}
I20260619 11:08:57.146481 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c27e5d4e5146419a542b958b087854" candidate_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" is_pre_election: true
I20260619 11:08:57.146785 2576 raft_consensus.cc:2454] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f62445c2d946b3a7c3f8d7519a8bea in term 0.
I20260619 11:08:57.147312 2643 leader_election.cc:302] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f62445c2d946b3a7c3f8d7519a8bea, 60bffce0294c4234b3ac7b942ac8c1a4; no voters:
I20260619 11:08:57.147585 2780 raft_consensus.cc:2781] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:08:57.147557 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c27e5d4e5146419a542b958b087854" candidate_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f963320f7c4268a290051efddd61fb" is_pre_election: true
I20260619 11:08:57.147697 2780 raft_consensus.cc:482] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:08:57.147751 2780 raft_consensus.cc:3037] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:57.147897 2442 raft_consensus.cc:2454] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f62445c2d946b3a7c3f8d7519a8bea in term 0.
I20260619 11:08:57.148900 2780 raft_consensus.cc:504] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.149256 2780 leader_election.cc:288] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [CANDIDATE]: Term 1 election: Requested vote from peers 60bffce0294c4234b3ac7b942ac8c1a4 (127.16.61.2:39891), e5f963320f7c4268a290051efddd61fb (127.16.61.1:34803)
I20260619 11:08:57.149609 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c27e5d4e5146419a542b958b087854" candidate_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60bffce0294c4234b3ac7b942ac8c1a4"
I20260619 11:08:57.149611 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c27e5d4e5146419a542b958b087854" candidate_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5f963320f7c4268a290051efddd61fb"
I20260619 11:08:57.149736 2442 raft_consensus.cc:3037] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:57.149736 2576 raft_consensus.cc:3037] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:57.150972 2442 raft_consensus.cc:2454] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f62445c2d946b3a7c3f8d7519a8bea in term 1.
I20260619 11:08:57.150972 2576 raft_consensus.cc:2454] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f62445c2d946b3a7c3f8d7519a8bea in term 1.
I20260619 11:08:57.151271 2643 leader_election.cc:302] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f62445c2d946b3a7c3f8d7519a8bea, 60bffce0294c4234b3ac7b942ac8c1a4; no voters:
I20260619 11:08:57.151432 2780 raft_consensus.cc:2781] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:57.151646 2780 raft_consensus.cc:686] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 1 LEADER]: Becoming Leader. State: Replica: 01f62445c2d946b3a7c3f8d7519a8bea, State: Running, Role: LEADER
I20260619 11:08:57.152038 2780 consensus_queue.cc:237] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } }
I20260619 11:08:57.154978 2297 catalog_manager.cc:5495] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea reported cstate change: term changed from 0 to 1, leader changed from <none> to 01f62445c2d946b3a7c3f8d7519a8bea (127.16.61.3). New cstate: current_term: 1 leader_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f62445c2d946b3a7c3f8d7519a8bea" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 33009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 } health_report { overall_health: UNKNOWN } } }
I20260619 11:08:57.166396 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.162991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.16.61.3:33009)
0619 11:08:57.163099 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.163110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.163122 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:08:57.163245 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.163248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.163294 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.163377 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:08:57.163580 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.164108 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.164972 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.165207 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.165364 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.165655 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:08:57.166014 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:08:57.166117 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.166211 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.166241 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:08:57.171890 2442 raft_consensus.cc:1261] T a1c27e5d4e5146419a542b958b087854 P e5f963320f7c4268a290051efddd61fb [term 1 FOLLOWER]: Refusing update from remote peer 01f62445c2d946b3a7c3f8d7519a8bea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:57.171969 2576 raft_consensus.cc:1261] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 1 FOLLOWER]: Refusing update from remote peer 01f62445c2d946b3a7c3f8d7519a8bea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:57.172618 2783 consensus_queue.cc:1035] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5f963320f7c4268a290051efddd61fb" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:57.172840 2780 consensus_queue.cc:1035] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [LEADER]: Connected to new peer: Peer: permanent_uuid: "60bffce0294c4234b3ac7b942ac8c1a4" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:57.182161 2788 mvcc.cc:205] Tried to move back new op lower bound from 7298528613048852480 to 7298528612977172480. Current Snapshot: MvccSnapshot[applied={T|T < 7298528613048852480}]
I20260619 11:08:57.189347 2297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:57.189486 2297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:57.191843 2297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c129586e4f3f43a0ae0c641c7dfa09f4: Sending DeleteTablet for 3 replicas of tablet a1c27e5d4e5146419a542b958b087854
I20260619 11:08:57.192607 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c27e5d4e5146419a542b958b087854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:57 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:57482
I20260619 11:08:57.192607 2422 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c27e5d4e5146419a542b958b087854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:57 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:50406
I20260619 11:08:57.193089 2797 tablet_replica.cc:330] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: stopping tablet replica
I20260619 11:08:57.193357 2690 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c27e5d4e5146419a542b958b087854 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:57 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:40688
I20260619 11:08:57.193429 2797 raft_consensus.cc:2229] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:57.193722 2797 raft_consensus.cc:2258] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:57.193768 2798 tablet_replica.cc:330] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: stopping tablet replica
I20260619 11:08:57.194059 2798 raft_consensus.cc:2229] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:08:57.194166 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 2359
I20260619 11:08:57.194460 2798 raft_consensus.cc:2258] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:57.195344 2797 ts_tablet_manager.cc:1905] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:57.195848 2798 ts_tablet_manager.cc:1905] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:57.198159 2797 ts_tablet_manager.cc:1918] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:57.198254 2797 log.cc:1196] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/a1c27e5d4e5146419a542b958b087854
I20260619 11:08:57.198634 2797 ts_tablet_manager.cc:1939] T a1c27e5d4e5146419a542b958b087854 P 60bffce0294c4234b3ac7b942ac8c1a4: Deleting consensus metadata
I20260619 11:08:57.199146 2798 ts_tablet_manager.cc:1918] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:57.199272 2798 log.cc:1196] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/a1c27e5d4e5146419a542b958b087854
I20260619 11:08:57.199621 2798 ts_tablet_manager.cc:1939] T a1c27e5d4e5146419a542b958b087854 P 01f62445c2d946b3a7c3f8d7519a8bea: Deleting consensus metadata
I20260619 11:08:57.199841 2281 catalog_manager.cc:4841] TS 60bffce0294c4234b3ac7b942ac8c1a4 (127.16.61.2:39891): tablet a1c27e5d4e5146419a542b958b087854 (table test-table [id=410e26edf7144e6585a22100586fc8e3]) successfully deleted
I20260619 11:08:57.200685 2283 catalog_manager.cc:4841] TS 01f62445c2d946b3a7c3f8d7519a8bea (127.16.61.3:33009): tablet a1c27e5d4e5146419a542b958b087854 (table test-table [id=410e26edf7144e6585a22100586fc8e3]) successfully deleted
W20260619 11:08:57.205289 2283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:34803 (error 108)
I20260619 11:08:57.205492 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 2493
I20260619 11:08:57.206895 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.206532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:34803 (local address 127.0.0.1:50414)
0619 11:08:57.206721 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.206796 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:34803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 11:08:57.207145 2283 catalog_manager.cc:4584] TS e5f963320f7c4268a290051efddd61fb (127.16.61.1:34803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1c27e5d4e5146419a542b958b087854: Network error: Client connection negotiation failed: client connection to 127.16.61.1:34803: connect: Connection refused (error 111)
I20260619 11:08:57.212168 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 2627
I20260619 11:08:57.217855 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 2265
2026-06-19T11:08:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2181 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T11:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:08:57Z Disabled control of system clock
I20260619 11:08:57.246821 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:43379
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:36301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:43379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:08:57.352239 2803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:57.352464 2803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:57.352497 2803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:57.356369 2803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:08:57.356415 2803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:57.356431 2803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:57.356447 2803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:08:57.356488 2803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:08:57.361119 2803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:43379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:43379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:57.362028 2803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:57.363305 2803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:57.371229 2808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:57.371198 2811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:57.371397 2803 server_base.cc:1034] running on GCE node
W20260619 11:08:57.371205 2809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:57.372151 2803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:57.373063 2803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:57.374233 2803 hybrid_clock.cc:648] HybridClock initialized: now 1781867337374203 us; error 40 us; skew 500 ppm
I20260619 11:08:57.375965 2803 webserver.cc:460] Webserver started at http://127.16.61.62:37543/ using document root <none> and password file <none>
I20260619 11:08:57.376452 2803 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:57.376507 2803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:57.376659 2803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:57.378487 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "4dd6d9c920b64423a4190f872cebc420"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "316e89256e109904bc4d0b64b464514b"
server_key_iv: "52af430a08b7178281e50cf2e27b4b79"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.378954 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "4dd6d9c920b64423a4190f872cebc420"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "316e89256e109904bc4d0b64b464514b"
server_key_iv: "52af430a08b7178281e50cf2e27b4b79"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.382396 2803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:08:57.384698 2817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.385854 2803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:57.385946 2803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "4dd6d9c920b64423a4190f872cebc420"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "316e89256e109904bc4d0b64b464514b"
server_key_iv: "52af430a08b7178281e50cf2e27b4b79"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.386025 2803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:57.400235 2803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:57.400790 2803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:57.400923 2803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:57.407743 2803 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:43379
I20260619 11:08:57.407768 2869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:43379 every 8 connection(s)
I20260619 11:08:57.408608 2803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:08:57.411546 2870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:57.411796 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 2803
I20260619 11:08:57.411927 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:08:57.412225 16628 external_mini_cluster.cc:1428] Setting key 1b44a30f443ab32e9667214e9e4e7b61
I20260619 11:08:57.417429 2870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420: Bootstrap starting.
I20260619 11:08:57.419956 2870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:57.420706 2870 log.cc:824] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:57.421139 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.413603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.16.61.62:43379)
0619 11:08:57.414170 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.414185 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.414220 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:08:57.414861 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.414876 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.415233 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.415489 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:57.415906 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.416826 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.417979 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.418423 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.418537 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.419355 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:57.419360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:57.419371 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:57.419800 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:57.419851 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:57.419871 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:57.419945 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.420128 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.420400 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":132,"threads_started":1}
I20260619 11:08:57.423487 2870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420: No bootstrap required, opened a new log
I20260619 11:08:57.426738 2870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd6d9c920b64423a4190f872cebc420" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43379 } }
I20260619 11:08:57.426945 2870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:57.427021 2870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd6d9c920b64423a4190f872cebc420, State: Initialized, Role: FOLLOWER
I20260619 11:08:57.427497 2870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd6d9c920b64423a4190f872cebc420" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43379 } }
I20260619 11:08:57.427637 2870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:08:57.427716 2870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:08:57.427840 2870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:57.428771 2870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd6d9c920b64423a4190f872cebc420" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43379 } }
I20260619 11:08:57.429128 2870 leader_election.cc:302] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dd6d9c920b64423a4190f872cebc420; no voters:
I20260619 11:08:57.429430 2870 leader_election.cc:288] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:08:57.429648 2875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:57.429905 2875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [term 1 LEADER]: Becoming Leader. State: Replica: 4dd6d9c920b64423a4190f872cebc420, State: Running, Role: LEADER
I20260619 11:08:57.430191 2875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd6d9c920b64423a4190f872cebc420" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43379 } }
I20260619 11:08:57.430616 2870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:08:57.431752 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dd6d9c920b64423a4190f872cebc420. Latest consensus state: current_term: 1 leader_uuid: "4dd6d9c920b64423a4190f872cebc420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd6d9c920b64423a4190f872cebc420" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43379 } } }
I20260619 11:08:57.431882 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:57.432255 2884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:08:57.432602 2876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dd6d9c920b64423a4190f872cebc420" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd6d9c920b64423a4190f872cebc420" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 43379 } } }
I20260619 11:08:57.432783 2876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420 [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:57.434715 2884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:08:57.439899 2884 catalog_manager.cc:1347] Generated new cluster ID: 555d30c71b324a908bfffe345d5a5fda
I20260619 11:08:57.439968 2884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:08:57.460566 2884 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:08:57.461496 2884 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:08:57.465680 2884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420: Generated new TSK 0
I20260619 11:08:57.466276 2884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:08:57.472911 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43379
--builtin_ntp_servers=127.16.61.20:36301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:08:57.583003 2894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:57.583233 2894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:57.583269 2894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:57.587183 2894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:57.587232 2894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:57.587284 2894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:08:57.592083 2894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:57.592967 2894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:57.594194 2894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:57.603302 2902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:57.603403 2899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:57.603479 2894 server_base.cc:1034] running on GCE node
W20260619 11:08:57.603713 2900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:57.604133 2894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:57.604717 2894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:57.605895 2894 hybrid_clock.cc:648] HybridClock initialized: now 1781867337605880 us; error 28 us; skew 500 ppm
I20260619 11:08:57.607570 2894 webserver.cc:460] Webserver started at http://127.16.61.1:46865/ using document root <none> and password file <none>
I20260619 11:08:57.608135 2894 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:57.608227 2894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:57.608435 2894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:57.610318 2894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "073ec5677c9b4a4497b235841f92e338"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "20b496c9c8a1555ec02b42569ddc289f"
server_key_iv: "7293e821495a1a64942beedaeb91fb21"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.610864 2894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "073ec5677c9b4a4497b235841f92e338"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "20b496c9c8a1555ec02b42569ddc289f"
server_key_iv: "7293e821495a1a64942beedaeb91fb21"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.614382 2894 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:08:57.616694 2908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.617815 2894 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:57.617940 2894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "073ec5677c9b4a4497b235841f92e338"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "20b496c9c8a1555ec02b42569ddc289f"
server_key_iv: "7293e821495a1a64942beedaeb91fb21"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.618045 2894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:57.637517 2894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:57.638221 2894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:57.638428 2894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:57.639066 2894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:57.640100 2894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:57.640177 2894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.640255 2894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:57.640298 2894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.648910 2894 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:44707
I20260619 11:08:57.648938 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:44707 every 8 connection(s)
I20260619 11:08:57.649758 2894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:08:57.658150 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.651492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:44579 (local address 127.16.61.62:43379)
0619 11:08:57.651631 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.651635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.652766 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 11:08:57.654077 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.654080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.654126 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.654206 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:57.655003 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.655512 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.656581 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.656791 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.656866 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.657774 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:57.657777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:57.657779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:57.657789 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:57.657807 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:57.657814 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:57.657872 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.657999 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.658047 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:08:57.658949 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.651894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43379 (local address 127.16.61.1:44579)
0619 11:08:57.652602 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.652655 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:57.653902 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:57.654356 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:57.654362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:57.654441 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:57.654888 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.654899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.655615 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.655618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.656484 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.656492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.656909 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.656912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.657037 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.657549 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:08:57.657550 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:57.657586 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:08:57.657643 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:57.657886 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:57.657892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:57.657895 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 11:08:57.657984 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 11:08:57.658174 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
I20260619 11:08:57.659402 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 2894
I20260619 11:08:57.659531 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:08:57.659889 16628 external_mini_cluster.cc:1428] Setting key 0a9ebce3e28b7f74ea01687cb7f602b5
I20260619 11:08:57.660228 3022 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43379
I20260619 11:08:57.660452 3022 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:57.660930 3022 heartbeater.cc:511] Master 127.16.61.62:43379 requested a full tablet report, sending...
I20260619 11:08:57.661774 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43379
--builtin_ntp_servers=127.16.61.20:36301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:08:57.662465 2834 ts_manager.cc:194] Registered new tserver with Master: 073ec5677c9b4a4497b235841f92e338 (127.16.61.1:44707)
I20260619 11:08:57.663806 2834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.1:44579
W20260619 11:08:57.767365 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:57.767576 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:57.767611 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:57.771466 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:57.771509 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:57.771559 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:08:57.776015 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:57.776942 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:57.778110 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:57.787003 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:57.787083 3025 server_base.cc:1034] running on GCE node
W20260619 11:08:57.787191 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:57.787015 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:57.787639 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:57.788218 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:57.789414 3025 hybrid_clock.cc:648] HybridClock initialized: now 1781867337789401 us; error 30 us; skew 500 ppm
I20260619 11:08:57.791038 3025 webserver.cc:460] Webserver started at http://127.16.61.2:40233/ using document root <none> and password file <none>
I20260619 11:08:57.791522 3025 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:57.791577 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:57.791805 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:57.793606 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "0c594cd1c4ff411eb062ee5385574101"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "22fd21ddccff0dad92db5c527aaf594a"
server_key_iv: "194de43941c0e05a48956b41d0c873c7"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.794078 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "0c594cd1c4ff411eb062ee5385574101"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "22fd21ddccff0dad92db5c527aaf594a"
server_key_iv: "194de43941c0e05a48956b41d0c873c7"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.797420 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:08:57.799753 3039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.800859 3025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:57.800983 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "0c594cd1c4ff411eb062ee5385574101"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "22fd21ddccff0dad92db5c527aaf594a"
server_key_iv: "194de43941c0e05a48956b41d0c873c7"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.801102 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:57.810397 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:57.810956 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:57.811143 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:57.811640 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:57.812529 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:57.812600 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.812666 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:57.812718 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.821144 3025 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:44307
I20260619 11:08:57.821174 3152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:44307 every 8 connection(s)
I20260619 11:08:57.821979 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:08:57.827755 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 3025
I20260619 11:08:57.827886 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:08:57.828143 16628 external_mini_cluster.cc:1428] Setting key 08d70bf7e6d52787b8f1767850857360
I20260619 11:08:57.830230 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43379
--builtin_ntp_servers=127.16.61.20:36301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:08:57.830852 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.823591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:51043 (local address 127.16.61.62:43379)
0619 11:08:57.823751 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:57.823755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:57.824767 (+ 1012us) server_negotiation.cc:407] Connection header received
0619 11:08:57.826023 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:57.826025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:57.826070 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:57.826139 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:57.826892 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.827410 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.828750 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.828997 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.829113 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.830256 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:57.830259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:57.830263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:57.830276 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:57.830299 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:57.830309 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:57.830389 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:57.830689 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 11:08:57.830749 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 11:08:57.832003 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:57.823988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43379 (local address 127.16.61.2:51043)
0619 11:08:57.824603 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:57.824663 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:57.825850 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:57.826261 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:57.826267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:57.826345 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:57.826779 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.826790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.827529 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.827532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.828576 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:57.828586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:57.829176 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:57.829180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:57.829397 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:57.830012 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:08:57.830014 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:57.830048 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:08:57.830105 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:57.830435 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:57.830478 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:57.830488 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 11:08:57.830646 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 11:08:57.830929 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260619 11:08:57.833174 3153 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43379
I20260619 11:08:57.833380 3153 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:57.833822 3153 heartbeater.cc:511] Master 127.16.61.62:43379 requested a full tablet report, sending...
I20260619 11:08:57.834738 2834 ts_manager.cc:194] Registered new tserver with Master: 0c594cd1c4ff411eb062ee5385574101 (127.16.61.2:44307)
I20260619 11:08:57.835280 2834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.2:51043
W20260619 11:08:57.937181 3156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:57.937399 3156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:57.937434 3156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:57.941339 3156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:57.941381 3156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:57.941474 3156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:08:57.946051 3156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:36301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:43379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:57.946918 3156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:57.948259 3156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:57.956943 3161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:57.956969 3162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:57.956943 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:57.957257 3156 server_base.cc:1034] running on GCE node
I20260619 11:08:57.957561 3156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:57.958115 3156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:57.959319 3156 hybrid_clock.cc:648] HybridClock initialized: now 1781867337959286 us; error 43 us; skew 500 ppm
I20260619 11:08:57.961162 3156 webserver.cc:460] Webserver started at http://127.16.61.3:40539/ using document root <none> and password file <none>
I20260619 11:08:57.961681 3156 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:57.961741 3156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:57.961931 3156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:57.963836 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "3cdaffa4fa734a11b347951c23adce4d"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "bbb6e48d412c1f4fec5e779b17692aa6"
server_key_iv: "76aaac68b7179811067d0f027a7fe318"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.964370 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "3cdaffa4fa734a11b347951c23adce4d"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "bbb6e48d412c1f4fec5e779b17692aa6"
server_key_iv: "76aaac68b7179811067d0f027a7fe318"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.967788 3156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:08:57.969998 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.971220 3156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:08:57.971339 3156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "3cdaffa4fa734a11b347951c23adce4d"
format_stamp: "Formatted at 2026-06-19 11:08:57 on dist-test-slave-8vcz"
server_key: "bbb6e48d412c1f4fec5e779b17692aa6"
server_key_iv: "76aaac68b7179811067d0f027a7fe318"
server_key_version: "encryptionkey@0"
I20260619 11:08:57.971431 3156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:57.989526 3156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:57.990139 3156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:57.990314 3156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:57.990774 3156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:57.991637 3156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:57.991732 3156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:57.991773 3156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:57.991794 3156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:58.000232 3156 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:36795
I20260619 11:08:58.000257 3283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:36795 every 8 connection(s)
I20260619 11:08:58.001058 3156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:08:58.006515 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 3156
I20260619 11:08:58.006604 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:08:58.006877 16628 external_mini_cluster.cc:1428] Setting key 919ccea76b063565c6745db13d43008c
I20260619 11:08:58.010046 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.002695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:60139 (local address 127.16.61.62:43379)
0619 11:08:58.002857 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:58.002862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:58.003922 (+ 1060us) server_negotiation.cc:407] Connection header received
0619 11:08:58.005403 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:58.005407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:58.005466 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:58.005545 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:58.006345 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.007005 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.008351 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.008579 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.008662 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.009687 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:58.009690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:58.009692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:58.009703 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:58.009721 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:58.009729 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:58.009786 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:58.009893 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 11:08:58.009936 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 11:08:58.010831 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.003098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:43379 (local address 127.16.61.3:60139)
0619 11:08:58.003778 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:58.003815 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:58.005174 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:58.005686 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:58.005692 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:58.005774 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:58.006196 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.006207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.007135 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.007138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.008236 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.008245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.008722 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.008726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.008921 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.009464 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:08:58.009466 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:58.009501 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:08:58.009555 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:58.009801 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:58.009806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:58.009810 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:08:58.009896 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 11:08:58.010088 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
I20260619 11:08:58.011933 3284 heartbeater.cc:347] Connected to a master server at 127.16.61.62:43379
I20260619 11:08:58.012147 3284 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:58.012596 3284 heartbeater.cc:511] Master 127.16.61.62:43379 requested a full tablet report, sending...
I20260619 11:08:58.013514 2834 ts_manager.cc:194] Registered new tserver with Master: 3cdaffa4fa734a11b347951c23adce4d (127.16.61.3:36795)
I20260619 11:08:58.014051 2834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.3:60139
I20260619 11:08:58.020699 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:08:58.026364 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.022636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.16.61.62:43379)
0619 11:08:58.022771 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:58.022775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:58.022919 (+ 144us) server_negotiation.cc:407] Connection header received
0619 11:08:58.023042 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:58.023045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:58.023088 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:58.023193 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:58.023453 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.023988 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.024830 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.025035 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.025140 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.025893 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:58.025896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:58.025901 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:08:58.025916 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:58.025931 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:08:58.025938 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:58.026067 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:58.026198 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 11:08:58.026232 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 11:08:58.028968 2834 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:08:58.031932 2834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:08:58.045332 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.040639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.16.61.3:36795)
0619 11:08:58.041036 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:58.041044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:58.041060 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:08:58.041117 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:58.041120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:58.041289 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:58.041572 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:58.041796 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.042886 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.044236 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.044736 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.044861 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.045027 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:58.045079 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 11:08:58.045170 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":128,"threads_started":1}
I20260619 11:08:58.045323 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.040275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:36795 (local address 127.0.0.1:55400)
0619 11:08:58.040735 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:58.040766 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:58.040847 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:58.041434 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:58.041437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:58.041452 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:58.041662 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.041674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.042983 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.042987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.044128 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.044135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.044865 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.044869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.044949 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.044970 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 11:08:58.045053 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 11:08:58.045174 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":89,"spinlock_wait_cycles":3840,"thread_start_us":91,"threads_started":1}
I20260619 11:08:58.045428 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.039691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.16.61.2:44307)
0619 11:08:58.040963 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:58.040974 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:58.040994 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:08:58.041054 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:58.041058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:58.041223 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:58.041344 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:58.041777 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.042500 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.044110 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.044698 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.044823 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.045004 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:58.045177 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 11:08:58.045263 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":85,"threads_started":1}
I20260619 11:08:58.046038 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.039569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:44307 (local address 127.0.0.1:56676)
0619 11:08:58.040345 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:58.040392 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:58.040516 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:58.041442 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:58.041447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:58.041466 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:58.041634 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.041641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.042615 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.042622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.043978 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.043989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.044824 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.044827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.044925 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.044960 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 11:08:58.045077 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 11:08:58.045913 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":793,"spinlock_wait_cycles":11904,"thread_start_us":78,"threads_started":1}
I20260619 11:08:58.046463 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.039643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:44707 (local address 127.0.0.1:44626)
0619 11:08:58.040348 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:58.040395 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:58.040517 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:58.041289 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:58.041295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:58.041318 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:58.041469 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.041476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.043050 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.043053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.044110 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.044120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.046047 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.046051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.046170 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.046193 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 11:08:58.046255 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 11:08:58.046323 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":4,"spinlock_wait_cycles":7808,"thread_start_us":97,"threads_started":1}
I20260619 11:08:58.046962 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.039851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.16.61.1:44707)
0619 11:08:58.040805 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:58.040819 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:58.040844 (+ 25us) server_negotiation.cc:407] Connection header received
0619 11:08:58.040920 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:58.040925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:58.041121 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:58.041276 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:58.041617 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.042801 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.045077 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.045923 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.046046 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.046634 (+ 588us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:58.046714 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 11:08:58.046808 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":112,"threads_started":1}
I20260619 11:08:58.048396 3218 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1dd621ace64d03976f722c53f1698c (DEFAULT_TABLE table=test-table [id=8fc044c15a6e4703890443065f274b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:58.049073 2956 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1dd621ace64d03976f722c53f1698c (DEFAULT_TABLE table=test-table [id=8fc044c15a6e4703890443065f274b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:58.049467 3218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1dd621ace64d03976f722c53f1698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:58.049131 3087 tablet_service.cc:1467] Processing CreateTablet for tablet 5b1dd621ace64d03976f722c53f1698c (DEFAULT_TABLE table=test-table [id=8fc044c15a6e4703890443065f274b90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:08:58.049885 2956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1dd621ace64d03976f722c53f1698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:58.050042 3087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b1dd621ace64d03976f722c53f1698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:58.055833 3301 tablet_bootstrap.cc:492] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338: Bootstrap starting.
I20260619 11:08:58.056866 3303 tablet_bootstrap.cc:492] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: Bootstrap starting.
I20260619 11:08:58.057854 3301 tablet_bootstrap.cc:654] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:58.058490 3303 tablet_bootstrap.cc:654] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:58.058550 3301 log.cc:824] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:58.059159 3303 log.cc:824] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:58.059828 3302 tablet_bootstrap.cc:492] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: Bootstrap starting.
I20260619 11:08:58.060417 3301 tablet_bootstrap.cc:492] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338: No bootstrap required, opened a new log
I20260619 11:08:58.060592 3301 ts_tablet_manager.cc:1397] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:08:58.061031 3303 tablet_bootstrap.cc:492] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: No bootstrap required, opened a new log
I20260619 11:08:58.061236 3303 ts_tablet_manager.cc:1397] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 11:08:58.061657 3302 tablet_bootstrap.cc:654] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:58.062284 3302 log.cc:824] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:58.063804 3302 tablet_bootstrap.cc:492] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: No bootstrap required, opened a new log
I20260619 11:08:58.063979 3302 ts_tablet_manager.cc:1397] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 11:08:58.064826 3301 raft_consensus.cc:348] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.065078 3301 raft_consensus.cc:374] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:58.065132 3301 raft_consensus.cc:729] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073ec5677c9b4a4497b235841f92e338, State: Initialized, Role: FOLLOWER
I20260619 11:08:58.065235 3303 raft_consensus.cc:348] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.065456 3303 raft_consensus.cc:374] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:58.065507 3303 raft_consensus.cc:729] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdaffa4fa734a11b347951c23adce4d, State: Initialized, Role: FOLLOWER
I20260619 11:08:58.065563 3301 consensus_queue.cc:260] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.065878 3303 consensus_queue.cc:260] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.066469 3301 ts_tablet_manager.cc:1428] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 11:08:58.066483 3284 heartbeater.cc:503] Master 127.16.61.62:43379 was elected leader, sending a full tablet report...
I20260619 11:08:58.066718 3303 ts_tablet_manager.cc:1428] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 11:08:58.066782 3022 heartbeater.cc:503] Master 127.16.61.62:43379 was elected leader, sending a full tablet report...
I20260619 11:08:58.068562 3302 raft_consensus.cc:348] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.068778 3302 raft_consensus.cc:374] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:58.068835 3302 raft_consensus.cc:729] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c594cd1c4ff411eb062ee5385574101, State: Initialized, Role: FOLLOWER
I20260619 11:08:58.069204 3302 consensus_queue.cc:260] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.069993 3153 heartbeater.cc:503] Master 127.16.61.62:43379 was elected leader, sending a full tablet report...
I20260619 11:08:58.070125 3302 ts_tablet_manager.cc:1428] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 11:08:58.072844 3154 tablet.cc:2369] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:08:58.107932 3307 raft_consensus.cc:482] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:08:58.108167 3307 raft_consensus.cc:504] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.109256 3307 leader_election.cc:288] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c594cd1c4ff411eb062ee5385574101 (127.16.61.2:44307), 3cdaffa4fa734a11b347951c23adce4d (127.16.61.3:36795)
I20260619 11:08:58.113131 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.109533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:55459 (local address 127.16.61.2:44307)
0619 11:08:58.109673 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:58.109677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:58.109692 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:08:58.109875 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:58.109880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:58.109935 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:58.110014 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:58.110484 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.111144 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.112213 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.112592 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.112711 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.112779 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:58.112959 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 11:08:58.113026 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:08:58.113134 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.109427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:44307 (local address 127.16.61.1:55459)
0619 11:08:58.109597 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:58.109615 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:58.109712 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:58.110144 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:58.110148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:58.110172 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:58.110335 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.110345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.111266 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.111270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.112092 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.112101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.112713 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.112716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.112829 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.112847 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:08:58.112932 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 11:08:58.113005 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"spinlock_wait_cycles":6400}
I20260619 11:08:58.113351 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.109872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:39897 (local address 127.16.61.3:36795)
0619 11:08:58.109991 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:58.109996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:58.110164 (+ 168us) server_negotiation.cc:407] Connection header received
0619 11:08:58.110299 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:58.110304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:58.110353 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:58.110424 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:08:58.110818 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.111311 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.112419 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.112804 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.112917 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.112974 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:58.113179 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 11:08:58.113244 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:08:58.113370 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.109690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:36795 (local address 127.16.61.1:39897)
0619 11:08:58.110052 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:58.110067 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:58.110204 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:58.110443 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:58.110445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:58.110460 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:08:58.110668 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.110675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.111424 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.111429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.112295 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:58.112303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.112943 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.112947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:58.113054 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.113071 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 11:08:58.113165 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 11:08:58.113228 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":10,"thread_start_us":118,"threads_started":1}
I20260619 11:08:58.113803 3107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1dd621ace64d03976f722c53f1698c" candidate_uuid: "073ec5677c9b4a4497b235841f92e338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c594cd1c4ff411eb062ee5385574101" is_pre_election: true
I20260619 11:08:58.114017 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1dd621ace64d03976f722c53f1698c" candidate_uuid: "073ec5677c9b4a4497b235841f92e338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cdaffa4fa734a11b347951c23adce4d" is_pre_election: true
I20260619 11:08:58.114099 3107 raft_consensus.cc:2454] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 073ec5677c9b4a4497b235841f92e338 in term 0.
I20260619 11:08:58.114260 3238 raft_consensus.cc:2454] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 073ec5677c9b4a4497b235841f92e338 in term 0.
I20260619 11:08:58.114586 2911 leader_election.cc:302] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 073ec5677c9b4a4497b235841f92e338, 0c594cd1c4ff411eb062ee5385574101; no voters:
I20260619 11:08:58.114871 3307 raft_consensus.cc:2781] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:08:58.114974 3307 raft_consensus.cc:482] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:08:58.115036 3307 raft_consensus.cc:3037] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:58.116048 3307 raft_consensus.cc:504] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.116377 3307 leader_election.cc:288] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [CANDIDATE]: Term 1 election: Requested vote from peers 0c594cd1c4ff411eb062ee5385574101 (127.16.61.2:44307), 3cdaffa4fa734a11b347951c23adce4d (127.16.61.3:36795)
I20260619 11:08:58.116674 3107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1dd621ace64d03976f722c53f1698c" candidate_uuid: "073ec5677c9b4a4497b235841f92e338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c594cd1c4ff411eb062ee5385574101"
I20260619 11:08:58.116796 3238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b1dd621ace64d03976f722c53f1698c" candidate_uuid: "073ec5677c9b4a4497b235841f92e338" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cdaffa4fa734a11b347951c23adce4d"
I20260619 11:08:58.116822 3107 raft_consensus.cc:3037] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:58.116911 3238 raft_consensus.cc:3037] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:58.118067 3238 raft_consensus.cc:2454] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 073ec5677c9b4a4497b235841f92e338 in term 1.
I20260619 11:08:58.118067 3107 raft_consensus.cc:2454] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 073ec5677c9b4a4497b235841f92e338 in term 1.
I20260619 11:08:58.118357 2911 leader_election.cc:302] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 073ec5677c9b4a4497b235841f92e338, 0c594cd1c4ff411eb062ee5385574101; no voters:
I20260619 11:08:58.118525 3307 raft_consensus.cc:2781] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:58.118755 3307 raft_consensus.cc:686] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 1 LEADER]: Becoming Leader. State: Replica: 073ec5677c9b4a4497b235841f92e338, State: Running, Role: LEADER
I20260619 11:08:58.119113 3307 consensus_queue.cc:237] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } }
I20260619 11:08:58.122319 2833 catalog_manager.cc:5495] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 reported cstate change: term changed from 0 to 1, leader changed from <none> to 073ec5677c9b4a4497b235841f92e338 (127.16.61.1). New cstate: current_term: 1 leader_uuid: "073ec5677c9b4a4497b235841f92e338" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "073ec5677c9b4a4497b235841f92e338" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 44707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 } health_report { overall_health: UNKNOWN } } }
I20260619 11:08:58.143283 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.138755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.16.61.1:44707)
0619 11:08:58.138880 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:58.138884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:58.138896 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:08:58.138955 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:58.138957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:58.139009 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:58.139128 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:08:58.139367 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.140075 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.141352 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:58.141733 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:58.141843 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:58.142312 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:08:58.142917 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:08:58.143050 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:58.143142 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 11:08:58.143174 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:08:58.149214 3238 raft_consensus.cc:1261] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 1 FOLLOWER]: Refusing update from remote peer 073ec5677c9b4a4497b235841f92e338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:58.149199 3107 raft_consensus.cc:1261] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 1 FOLLOWER]: Refusing update from remote peer 073ec5677c9b4a4497b235841f92e338: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:08:58.149920 3311 consensus_queue.cc:1035] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cdaffa4fa734a11b347951c23adce4d" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 36795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:08:58.150104 3307 consensus_queue.cc:1035] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c594cd1c4ff411eb062ee5385574101" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 44307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 11:08:58.150736 3023 tablet.cc:2369] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:08:58.157001 3314 mvcc.cc:205] Tried to move back new op lower bound from 7298528617051484160 to 7298528616938991616. Current Snapshot: MvccSnapshot[applied={T|T < 7298528617051484160}]
I20260619 11:08:58.158018 3316 mvcc.cc:205] Tried to move back new op lower bound from 7298528617051484160 to 7298528616938991616. Current Snapshot: MvccSnapshot[applied={T|T < 7298528617051484160}]
I20260619 11:08:58.166601 2834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:58.166746 2834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:08:58.169025 2834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dd6d9c920b64423a4190f872cebc420: Sending DeleteTablet for 3 replicas of tablet 5b1dd621ace64d03976f722c53f1698c
I20260619 11:08:58.169728 2956 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1dd621ace64d03976f722c53f1698c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:58 UTC) from {username='slave'} at 127.0.0.1:44626
I20260619 11:08:58.169910 3087 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1dd621ace64d03976f722c53f1698c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:58 UTC) from {username='slave'} at 127.0.0.1:56676
I20260619 11:08:58.169921 3218 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b1dd621ace64d03976f722c53f1698c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:08:58 UTC) from {username='slave'} at 127.0.0.1:55400
I20260619 11:08:58.170181 3324 tablet_replica.cc:330] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338: stopping tablet replica
I20260619 11:08:58.170444 3324 raft_consensus.cc:2229] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:08:58.170816 3324 raft_consensus.cc:2258] T 5b1dd621ace64d03976f722c53f1698c P 073ec5677c9b4a4497b235841f92e338 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:58.170908 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 2894
I20260619 11:08:58.173089 3325 tablet_replica.cc:330] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: stopping tablet replica
I20260619 11:08:58.173364 3325 raft_consensus.cc:2229] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:58.173635 3325 raft_consensus.cc:2258] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:58.174026 3326 tablet_replica.cc:330] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: stopping tablet replica
I20260619 11:08:58.174274 3326 raft_consensus.cc:2229] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:08:58.174543 3326 raft_consensus.cc:2258] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:08:58.175647 3326 ts_tablet_manager.cc:1905] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:08:58.175647 3325 ts_tablet_manager.cc:1905] T 5b1dd621ace64d03976f722c53f1698c P 0c594cd1c4ff411eb062ee5385574101: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 11:08:58.177585 2821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:44707 (error 108)
I20260619 11:08:58.177713 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 3025
I20260619 11:08:58.179018 3326 ts_tablet_manager.cc:1918] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:08:58.179081 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.178765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:44707 (local address 127.0.0.1:44644)
0619 11:08:58.178907 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:58.178983 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 11:08:58.179124 3326 log.cc:1196] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/5b1dd621ace64d03976f722c53f1698c
W20260619 11:08:58.179245 2821 catalog_manager.cc:4584] TS 073ec5677c9b4a4497b235841f92e338 (127.16.61.1:44707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b1dd621ace64d03976f722c53f1698c: Network error: Client connection negotiation failed: client connection to 127.16.61.1:44707: connect: Connection refused (error 111)
I20260619 11:08:58.179450 3326 ts_tablet_manager.cc:1939] T 5b1dd621ace64d03976f722c53f1698c P 3cdaffa4fa734a11b347951c23adce4d: Deleting consensus metadata
I20260619 11:08:58.180677 2820 catalog_manager.cc:4841] TS 3cdaffa4fa734a11b347951c23adce4d (127.16.61.3:36795): tablet 5b1dd621ace64d03976f722c53f1698c (table test-table [id=8fc044c15a6e4703890443065f274b90]) successfully deleted
I20260619 11:08:58.184377 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 3156
I20260619 11:08:58.185292 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:58.184499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:44307 (local address 127.0.0.1:56682)
0619 11:08:58.185189 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:58.185231 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.2:44307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613}
I20260619 11:08:58.190026 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 2803
2026-06-19T11:08:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:08:58 dist-test-slave-8vcz krb5kdc[3330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:08:58 dist-test-slave-8vcz krb5kdc[3330](info): set up 2 sockets
Jun 19 11:08:58 dist-test-slave-8vcz krb5kdc[3330](info): commencing operation
krb5kdc: starting...
W20260619 11:08:59.227528 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:08:59 dist-test-slave-8vcz krb5kdc[3330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:08:59.343056 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:33663
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:33663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:59.451465 3343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:59.451853 3343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:59.451946 3343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:59.459061 3343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:08:59.459159 3343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:59.459199 3343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:59.459245 3343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:08:59.459271 3343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:08:59.468230 3343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:33663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:33663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:59.469439 3343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:59.470758 3343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:59.481600 3348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:59.481685 3351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:59.481601 3349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:59.482039 3343 server_base.cc:1034] running on GCE node
I20260619 11:08:59.482571 3343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:59.483727 3343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:59.484930 3343 hybrid_clock.cc:648] HybridClock initialized: now 1781867339484897 us; error 48 us; skew 500 ppm
Jun 19 11:08:59 dist-test-slave-8vcz krb5kdc[3330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:59.488320 3343 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:08:59.489185 3343 webserver.cc:460] Webserver started at http://127.16.61.62:40371/ using document root <none> and password file <none>
I20260619 11:08:59.489728 3343 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:59.489822 3343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:59.490038 3343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:59.492218 3343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "6732f2fca34947469d7b35de2e1f5d90"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "c5a96ff98645c80cdeee9cccf0a4f610"
server_key_iv: "0a0d8d50af2afd1705cd18db99278c1e"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.492820 3343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "6732f2fca34947469d7b35de2e1f5d90"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "c5a96ff98645c80cdeee9cccf0a4f610"
server_key_iv: "0a0d8d50af2afd1705cd18db99278c1e"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.496253 3343 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:08:59.498479 3358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:59.499555 3343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:59.499734 3343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "6732f2fca34947469d7b35de2e1f5d90"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "c5a96ff98645c80cdeee9cccf0a4f610"
server_key_iv: "0a0d8d50af2afd1705cd18db99278c1e"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.499846 3343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:59.523597 3343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:59.526126 3343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:59.526309 3343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:59.532884 3343 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:33663
I20260619 11:08:59.532910 3410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:33663 every 8 connection(s)
I20260619 11:08:59.533746 3343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:08:59.536250 3411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:08:59.538520 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 3343
I20260619 11:08:59.538657 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:08:59.538959 16628 external_mini_cluster.cc:1428] Setting key ef8345d3ac6fe226f4c4b6e6da8edc3a
I20260619 11:08:59.541998 3411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90: Bootstrap starting.
I20260619 11:08:59.544073 3411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90: Neither blocks nor log segments found. Creating new log.
I20260619 11:08:59.544806 3411 log.cc:824] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90: Log is configured to *not* fsync() on all Append() calls
I20260619 11:08:59.546926 3411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90: No bootstrap required, opened a new log
Jun 19 11:08:59 dist-test-slave-8vcz krb5kdc[3330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:59.550964 3411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6732f2fca34947469d7b35de2e1f5d90" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33663 } }
I20260619 11:08:59.551275 3411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:08:59.551338 3411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6732f2fca34947469d7b35de2e1f5d90, State: Initialized, Role: FOLLOWER
I20260619 11:08:59.551821 3411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6732f2fca34947469d7b35de2e1f5d90" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33663 } }
I20260619 11:08:59.551973 3411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:08:59.552042 3411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:08:59.552137 3411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:08:59.553067 3411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6732f2fca34947469d7b35de2e1f5d90" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33663 } }
I20260619 11:08:59.553205 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:59.540306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.16.61.62:33663)
0619 11:08:59.540713 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:59.540726 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:59.540758 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:08:59.541352 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:59.541366 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:59.541672 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:59.541986 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:59.542777 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:59.543783 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:59.544679 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:59.545220 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:59.545375 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:59.548032 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:59.548038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:59.548055 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:59.548086 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:59.550460 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:59.550871 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:59.550877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:59.550882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:59.551019 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:59.551307 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:59.551311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:59.551314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:59.551791 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:59.552004 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:59.552270 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 11:08:59.552542 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20260619 11:08:59.553444 3411 leader_election.cc:302] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6732f2fca34947469d7b35de2e1f5d90; no voters:
I20260619 11:08:59.553785 3411 leader_election.cc:288] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:08:59.553857 3416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:08:59.554049 3416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [term 1 LEADER]: Becoming Leader. State: Replica: 6732f2fca34947469d7b35de2e1f5d90, State: Running, Role: LEADER
I20260619 11:08:59.554324 3416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6732f2fca34947469d7b35de2e1f5d90" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33663 } }
I20260619 11:08:59.556025 3418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6732f2fca34947469d7b35de2e1f5d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6732f2fca34947469d7b35de2e1f5d90" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33663 } } }
I20260619 11:08:59.556064 3417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6732f2fca34947469d7b35de2e1f5d90. Latest consensus state: current_term: 1 leader_uuid: "6732f2fca34947469d7b35de2e1f5d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6732f2fca34947469d7b35de2e1f5d90" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33663 } } }
I20260619 11:08:59.556144 3418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:59.556159 3411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:08:59.556190 3417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90 [sys.catalog]: This master's current role is: LEADER
I20260619 11:08:59.556499 3420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:08:59.559418 3420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:08:59.564707 3420 catalog_manager.cc:1347] Generated new cluster ID: dd19383374c844deabaa434220cb094b
I20260619 11:08:59.564775 3420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:08:59.577003 3420 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:08:59.577848 3420 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:08:59.587285 3420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90: Generated new TSK 0
I20260619 11:08:59.587899 3420 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.61.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
Entry for principal kudu/127.16.61.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab.
I20260619 11:08:59.618816 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:33663
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:59.724269 3437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:59.724489 3437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:59.724529 3437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:59.728489 3437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:59.728538 3437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:59.728618 3437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:08:59.733655 3437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.1.keytab
--principal=kudu/127.16.61.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:59.734556 3437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:59.735474 3437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:59.744345 3443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:59.744379 3445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:59.744366 3442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:59.744549 3437 server_base.cc:1034] running on GCE node
I20260619 11:08:59.745066 3437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:59.745656 3437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:59.746837 3437 hybrid_clock.cc:648] HybridClock initialized: now 1781867339746824 us; error 25 us; skew 500 ppm
Jun 19 11:08:59 dist-test-slave-8vcz krb5kdc[3330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:59.749436 3437 init.cc:377] Logged in from keytab as kudu/127.16.61.1@KRBTEST.COM (short username kudu)
I20260619 11:08:59.750250 3437 webserver.cc:460] Webserver started at http://127.16.61.1:43761/ using document root <none> and password file <none>
I20260619 11:08:59.750721 3437 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:59.750778 3437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:59.750924 3437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:59.752818 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "d8017f1c27b845d39a3da8a4b2146e39"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "df4731acc9317d56a0e084d3b3fc3a7b"
server_key_iv: "5ef88812f291c66fd1607db7c60cd83b"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.753300 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "d8017f1c27b845d39a3da8a4b2146e39"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "df4731acc9317d56a0e084d3b3fc3a7b"
server_key_iv: "5ef88812f291c66fd1607db7c60cd83b"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.756729 3437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:08:59.759141 3452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:59.760315 3437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:08:59.760426 3437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "d8017f1c27b845d39a3da8a4b2146e39"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "df4731acc9317d56a0e084d3b3fc3a7b"
server_key_iv: "5ef88812f291c66fd1607db7c60cd83b"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.760525 3437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:08:59.780790 3437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:08:59.783363 3437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:08:59.783555 3437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:08:59.784124 3437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:08:59.784893 3437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:08:59.784937 3437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:59.785005 3437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:08:59.785037 3437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:59.793442 3437 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:40365
I20260619 11:08:59.793468 3565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:40365 every 8 connection(s)
I20260619 11:08:59.794291 3437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
Jun 19 11:08:59 dist-test-slave-8vcz krb5kdc[3330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.1@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:08:59.804481 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 3437
I20260619 11:08:59.804641 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:08:59.804998 16628 external_mini_cluster.cc:1428] Setting key f56d1b86e31b577c8acaaef999d61051
I20260619 11:08:59.807569 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:59.796002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:47491 (local address 127.16.61.62:33663)
0619 11:08:59.796124 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:08:59.796128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:08:59.796895 (+ 767us) server_negotiation.cc:407] Connection header received
0619 11:08:59.798201 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:08:59.798204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:08:59.798262 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:08:59.798368 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:08:59.799424 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:59.800016 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:59.801103 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:08:59.801344 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:08:59.801454 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:59.803983 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:08:59.803986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:08:59.803988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:08:59.804012 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:08:59.805548 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:59.806002 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:59.806005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:59.806006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:59.806124 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:08:59.806532 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:08:59.806535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:08:59.806536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:08:59.806701 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:08:59.806777 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:08:59.807369 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 11:08:59.807460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:08:59.808593 3568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:08:59.796351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:33663 (local address 127.16.61.1:47491)
0619 11:08:59.796762 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 11:08:59.796799 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 11:08:59.797997 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:08:59.798534 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:08:59.798541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:08:59.798863 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:08:59.799284 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:59.799297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:59.800134 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:59.800138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:59.800980 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:08:59.800989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:08:59.801485 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:08:59.801489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:08:59.801654 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:08:59.802121 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:08:59.802136 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:08:59.803835 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:08:59.805636 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:59.805642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:59.805654 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:59.805905 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:59.806234 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:08:59.806238 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:08:59.806240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:08:59.806428 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:08:59.806813 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:08:59.806823 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:08:59.807125 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 11:08:59.807335 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 11:08:59.807704 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260619 11:08:59.809690 3566 heartbeater.cc:347] Connected to a master server at 127.16.61.62:33663
I20260619 11:08:59.809913 3566 heartbeater.cc:464] Registering TS with master...
I20260619 11:08:59.810458 3566 heartbeater.cc:511] Master 127.16.61.62:33663 requested a full tablet report, sending...
I20260619 11:08:59.812156 3375 ts_manager.cc:194] Registered new tserver with Master: d8017f1c27b845d39a3da8a4b2146e39 (127.16.61.1:40365)
I20260619 11:08:59.813215 3375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.1@KRBTEST.COM'} at 127.16.61.1:47491
WARNING: no policy specified for kudu/127.16.61.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
Entry for principal kudu/127.16.61.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab.
I20260619 11:08:59.830425 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:33663
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:08:59.935112 3571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:08:59.935333 3571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:08:59.935372 3571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:08:59.939246 3571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:08:59.939301 3571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:08:59.939354 3571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:08:59.944465 3571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.2.keytab
--principal=kudu/127.16.61.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:08:59.945327 3571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:08:59.946151 3571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:08:59.954882 3576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:08:59.954945 3579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:59.954959 3571 server_base.cc:1034] running on GCE node
W20260619 11:08:59.954890 3577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:08:59.955564 3571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:08:59.956197 3571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:08:59.957367 3571 hybrid_clock.cc:648] HybridClock initialized: now 1781867339957342 us; error 37 us; skew 500 ppm
Jun 19 11:08:59 dist-test-slave-8vcz krb5kdc[3330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:08:59.959914 3571 init.cc:377] Logged in from keytab as kudu/127.16.61.2@KRBTEST.COM (short username kudu)
I20260619 11:08:59.960697 3571 webserver.cc:460] Webserver started at http://127.16.61.2:43543/ using document root <none> and password file <none>
I20260619 11:08:59.961159 3571 fs_manager.cc:359] Metadata directory not provided
I20260619 11:08:59.961212 3571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:08:59.961369 3571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:08:59.963176 3571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "7467d94bad034ef7932208181e3e0fd2"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "0b11f5c7d80c7e1ca6e78ba8e42f31c5"
server_key_iv: "726e8762e7fc3731102514414e6c7c49"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.963794 3571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "7467d94bad034ef7932208181e3e0fd2"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "0b11f5c7d80c7e1ca6e78ba8e42f31c5"
server_key_iv: "726e8762e7fc3731102514414e6c7c49"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.967067 3571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:08:59.969230 3586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:08:59.970317 3571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 11:08:59.970405 3571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "7467d94bad034ef7932208181e3e0fd2"
format_stamp: "Formatted at 2026-06-19 11:08:59 on dist-test-slave-8vcz"
server_key: "0b11f5c7d80c7e1ca6e78ba8e42f31c5"
server_key_iv: "726e8762e7fc3731102514414e6c7c49"
server_key_version: "encryptionkey@0"
I20260619 11:08:59.970479 3571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:00.011970 3571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:00.014869 3571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:00.015025 3571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:00.015611 3571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:00.016497 3571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:00.016546 3571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.016585 3571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:00.016606 3571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.025046 3571 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:33635
I20260619 11:09:00.025113 3699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:33635 every 8 connection(s)
I20260619 11:09:00.026011 3571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
Jun 19 11:09:00 dist-test-slave-8vcz krb5kdc[3330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.2@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:00.036062 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 3571
I20260619 11:09:00.036173 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:00.036458 16628 external_mini_cluster.cc:1428] Setting key 213bdfedf22654368ccda182ce051bef
I20260619 11:09:00.039906 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.027828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:40481 (local address 127.16.61.62:33663)
0619 11:09:00.027963 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.027967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.028758 (+ 791us) server_negotiation.cc:407] Connection header received
0619 11:09:00.030102 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.030105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.030159 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.030241 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:00.031372 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.032036 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.033160 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.033396 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.033508 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.035893 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:00.035896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:00.035898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:00.035922 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:00.037601 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:00.038158 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:00.038161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:00.038162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:00.038240 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:00.038667 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:00.038670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:00.038671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:00.038845 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:00.038930 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.039520 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.039698 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:09:00.040716 3702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.028184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:33663 (local address 127.16.61.2:40481)
0619 11:09:00.028611 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.028654 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:00.029897 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:00.030409 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:00.030416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:00.030754 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:00.031224 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.031237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.032161 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.032165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.033032 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.033041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.033545 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.033549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.033718 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.034254 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:00.034271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:00.035658 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:00.037703 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:00.037709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:00.037729 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:00.038052 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:00.038341 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:00.038345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:00.038347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:00.038536 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:00.038955 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:00.038964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:00.039274 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 11:09:00.039485 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 11:09:00.039883 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
I20260619 11:09:00.041790 3700 heartbeater.cc:347] Connected to a master server at 127.16.61.62:33663
I20260619 11:09:00.042033 3700 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:00.042601 3700 heartbeater.cc:511] Master 127.16.61.62:33663 requested a full tablet report, sending...
I20260619 11:09:00.043748 3375 ts_manager.cc:194] Registered new tserver with Master: 7467d94bad034ef7932208181e3e0fd2 (127.16.61.2:33635)
I20260619 11:09:00.044297 3375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.2@KRBTEST.COM'} at 127.16.61.2:40481
WARNING: no policy specified for kudu/127.16.61.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
Entry for principal kudu/127.16.61.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab.
I20260619 11:09:00.062772 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:33663
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:00.170349 3705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:00.170576 3705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:00.170650 3705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:00.174559 3705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:00.174607 3705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:00.174706 3705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:09:00.179682 3705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.3.keytab
--principal=kudu/127.16.61.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:00.180636 3705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:00.181561 3705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:00.190634 3713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:00.190701 3711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:00.190699 3705 server_base.cc:1034] running on GCE node
W20260619 11:09:00.190649 3710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:00.191331 3705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:00.191923 3705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:00.193073 3705 hybrid_clock.cc:648] HybridClock initialized: now 1781867340193041 us; error 44 us; skew 500 ppm
Jun 19 11:09:00 dist-test-slave-8vcz krb5kdc[3330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867340, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:00.195529 3705 init.cc:377] Logged in from keytab as kudu/127.16.61.3@KRBTEST.COM (short username kudu)
I20260619 11:09:00.196318 3705 webserver.cc:460] Webserver started at http://127.16.61.3:36863/ using document root <none> and password file <none>
I20260619 11:09:00.196785 3705 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:00.196839 3705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:00.196987 3705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:00.198770 3705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "1bdceafbd6350d0549d5bb40770cbdd8"
server_key_iv: "0b52ebff4673cfb0e62620fb307e3792"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.199254 3705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "1bdceafbd6350d0549d5bb40770cbdd8"
server_key_iv: "0b52ebff4673cfb0e62620fb307e3792"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.202704 3705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:00.205184 3720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.206316 3705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:00.206411 3705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "1bdceafbd6350d0549d5bb40770cbdd8"
server_key_iv: "0b52ebff4673cfb0e62620fb307e3792"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.206490 3705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:00.221221 3705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:00.224241 3705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:00.224380 3705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:00.224833 3705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:00.225528 3705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:00.225571 3705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.225607 3705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:00.225627 3705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.233922 3705 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:42213
I20260619 11:09:00.233952 3833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:42213 every 8 connection(s)
I20260619 11:09:00.234726 3705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:09:00.238694 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 3705
I20260619 11:09:00.238843 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:00.239136 16628 external_mini_cluster.cc:1428] Setting key 31f6c0d1fc1f272f63ff916a5d2697f2
Jun 19 11:09:00 dist-test-slave-8vcz krb5kdc[3330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867340, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.3@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:00.247377 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.236421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:51925 (local address 127.16.61.62:33663)
0619 11:09:00.236559 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.236564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.237361 (+ 797us) server_negotiation.cc:407] Connection header received
0619 11:09:00.238668 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.238670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.238716 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.238815 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:00.240122 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.240644 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.241742 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.241959 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.242064 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.244287 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:00.244290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:00.244293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:00.244322 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:00.245483 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:00.245961 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:00.245966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:00.245968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:00.246045 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:00.246365 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:00.246370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:00.246372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:00.246538 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:00.246642 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.247095 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.247214 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:09:00.248111 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.236814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:33663 (local address 127.16.61.3:51925)
0619 11:09:00.237212 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.237250 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:00.238446 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:00.238997 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:00.239004 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:00.239381 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:00.239981 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.240003 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.240767 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.240770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.241615 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.241624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.242095 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.242099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.242270 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.242757 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:00.242772 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:00.244135 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:00.245595 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:00.245600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:00.245613 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:00.245858 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:00.246138 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:00.246141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:00.246143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:00.246270 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:00.246638 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:00.246644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:00.246867 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 11:09:00.247043 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 11:09:00.247327 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260619 11:09:00.249071 3834 heartbeater.cc:347] Connected to a master server at 127.16.61.62:33663
I20260619 11:09:00.249292 3834 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:00.249728 3834 heartbeater.cc:511] Master 127.16.61.62:33663 requested a full tablet report, sending...
I20260619 11:09:00.250659 3375 ts_manager.cc:194] Registered new tserver with Master: 02c39a1bf5b1479d8f5cc10d36e3ae54 (127.16.61.3:42213)
I20260619 11:09:00.251200 3375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.61.3@KRBTEST.COM'} at 127.16.61.3:51925
I20260619 11:09:00.252751 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:09:00.261263 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.254563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.16.61.62:33663)
0619 11:09:00.254691 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.254696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.254884 (+ 188us) server_negotiation.cc:407] Connection header received
0619 11:09:00.254962 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.254964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.254997 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.255058 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:00.255731 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.256197 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.257054 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.257252 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.257317 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.258653 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:00.258657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:00.258659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:00.258694 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:00.259935 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:00.260283 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:00.260286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:00.260287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:00.260352 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:00.260564 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:00.260566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:00.260567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:00.260751 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:00.260852 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.261078 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.261158 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:09:00.263859 3375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:09:00.266796 3375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:00.281504 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.276064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:40365 (local address 127.0.0.1:41438)
0619 11:09:00.276505 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.276515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:00.276580 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:00.277240 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:00.277243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:00.277260 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:00.277439 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.277449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.278929 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.278932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.280137 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.280146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.280939 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.280942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.281044 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.281078 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:09:00.281214 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 11:09:00.281337 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":54,"thread_start_us":94,"threads_started":1}
I20260619 11:09:00.281504 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.275367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:33635 (local address 127.0.0.1:46286)
0619 11:09:00.275899 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.275948 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:00.276064 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:00.276856 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:00.276863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:00.276888 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:00.277030 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.277037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.278569 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.278586 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.280034 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.280044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.281134 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.281138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.281264 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.281284 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 11:09:00.281335 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 11:09:00.281390 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":83,"thread_start_us":70,"threads_started":1}
I20260619 11:09:00.281651 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.275525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.16.61.2:33635)
0619 11:09:00.276350 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.276364 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.276388 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:09:00.276465 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.276471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.276673 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.277186 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:00.277587 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.278415 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.280157 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.280992 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.281184 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.281316 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.281389 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.281463 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":132,"threads_started":1}
I20260619 11:09:00.281951 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.275552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42213 (local address 127.0.0.1:46768)
0619 11:09:00.276052 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.276069 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:00.276177 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:00.277251 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:00.277253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:00.277274 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:00.277443 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.277454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.278888 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.278891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.280051 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.280064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.281134 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.281138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.281619 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.281641 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 11:09:00.281720 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 11:09:00.281807 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":429,"spinlock_wait_cycles":768,"thread_start_us":129,"threads_started":1}
I20260619 11:09:00.282366 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.276159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.16.61.3:42213)
0619 11:09:00.276840 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.276851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.276868 (+ 17us) server_negotiation.cc:407] Connection header received
0619 11:09:00.276928 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.276932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.277078 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.277397 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:00.277552 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.278754 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.280227 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.280992 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.281205 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.281316 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.282119 (+ 803us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.282226 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":154,"threads_started":1}
I20260619 11:09:00.283165 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.276482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.16.61.1:40365)
0619 11:09:00.276785 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.276799 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.276819 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:09:00.276900 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.276904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.277078 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.277566 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:00.277729 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.278819 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.280245 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.280810 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.280937 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.281088 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.282906 (+ 1818us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.283009 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260619 11:09:00.284891 3634 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4732fc01754500a0cb8c0de33dcb06 (DEFAULT_TABLE table=test-table [id=b2e1c6f47d654fccb12cc8d6c88010d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:00.285111 3768 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4732fc01754500a0cb8c0de33dcb06 (DEFAULT_TABLE table=test-table [id=b2e1c6f47d654fccb12cc8d6c88010d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:00.285903 3634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4732fc01754500a0cb8c0de33dcb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:00.285569 3500 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4732fc01754500a0cb8c0de33dcb06 (DEFAULT_TABLE table=test-table [id=b2e1c6f47d654fccb12cc8d6c88010d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:00.286101 3768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4732fc01754500a0cb8c0de33dcb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:00.286455 3500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4732fc01754500a0cb8c0de33dcb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:00.292881 3853 tablet_bootstrap.cc:492] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39: Bootstrap starting.
I20260619 11:09:00.293228 3851 tablet_bootstrap.cc:492] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: Bootstrap starting.
I20260619 11:09:00.295137 3853 tablet_bootstrap.cc:654] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:00.295595 3852 tablet_bootstrap.cc:492] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: Bootstrap starting.
I20260619 11:09:00.295652 3851 tablet_bootstrap.cc:654] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:00.296537 3851 log.cc:824] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:00.296537 3853 log.cc:824] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:00.297413 3852 tablet_bootstrap.cc:654] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:00.298545 3851 tablet_bootstrap.cc:492] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: No bootstrap required, opened a new log
I20260619 11:09:00.298545 3853 tablet_bootstrap.cc:492] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39: No bootstrap required, opened a new log
I20260619 11:09:00.298640 3852 log.cc:824] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:00.298730 3853 ts_tablet_manager.cc:1397] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 11:09:00.298730 3851 ts_tablet_manager.cc:1397] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 11:09:00.300283 3852 tablet_bootstrap.cc:492] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: No bootstrap required, opened a new log
I20260619 11:09:00.300469 3852 ts_tablet_manager.cc:1397] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:09:00.302644 3851 raft_consensus.cc:348] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.302642 3853 raft_consensus.cc:348] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.302925 3851 raft_consensus.cc:374] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:00.302925 3853 raft_consensus.cc:374] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:00.302985 3853 raft_consensus.cc:729] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8017f1c27b845d39a3da8a4b2146e39, State: Initialized, Role: FOLLOWER
I20260619 11:09:00.302994 3851 raft_consensus.cc:729] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7467d94bad034ef7932208181e3e0fd2, State: Initialized, Role: FOLLOWER
I20260619 11:09:00.303436 3853 consensus_queue.cc:260] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.303458 3851 consensus_queue.cc:260] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.303303 3852 raft_consensus.cc:348] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.303711 3852 raft_consensus.cc:374] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:00.303781 3852 raft_consensus.cc:729] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c39a1bf5b1479d8f5cc10d36e3ae54, State: Initialized, Role: FOLLOWER
I20260619 11:09:00.304186 3852 consensus_queue.cc:260] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.304437 3853 ts_tablet_manager.cc:1428] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 11:09:00.304437 3851 ts_tablet_manager.cc:1428] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260619 11:09:00.304790 3566 heartbeater.cc:503] Master 127.16.61.62:33663 was elected leader, sending a full tablet report...
I20260619 11:09:00.305104 3852 ts_tablet_manager.cc:1428] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 11:09:00.305414 3834 heartbeater.cc:503] Master 127.16.61.62:33663 was elected leader, sending a full tablet report...
I20260619 11:09:00.305575 3700 heartbeater.cc:503] Master 127.16.61.62:33663 was elected leader, sending a full tablet report...
I20260619 11:09:00.333688 3857 raft_consensus.cc:482] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:00.333865 3857 raft_consensus.cc:504] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.334897 3857 leader_election.cc:288] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7467d94bad034ef7932208181e3e0fd2 (127.16.61.2:33635), 02c39a1bf5b1479d8f5cc10d36e3ae54 (127.16.61.3:42213)
I20260619 11:09:00.338780 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.335251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:35381 (local address 127.16.61.3:42213)
0619 11:09:00.335394 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.335401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.335520 (+ 119us) server_negotiation.cc:407] Connection header received
0619 11:09:00.335603 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.335606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.335657 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.335752 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:00.336089 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.336770 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.337825 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.338227 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.338347 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.338408 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.338598 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.338665 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 11:09:00.338804 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.335118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:42213 (local address 127.16.61.1:35381)
0619 11:09:00.335425 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.335438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:00.335512 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:00.335800 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:00.335804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:00.335826 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:00.335967 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.335974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.336878 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.336884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.337694 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.337702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.338348 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.338352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.338447 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.338475 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 11:09:00.338578 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 11:09:00.338649 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":159,"threads_started":1}
I20260619 11:09:00.338904 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.335152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:45569 (local address 127.16.61.2:33635)
0619 11:09:00.335285 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.335291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.335509 (+ 218us) server_negotiation.cc:407] Connection header received
0619 11:09:00.335603 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.335606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.335657 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.335758 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:00.336245 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.336884 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.337901 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.338309 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.338427 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.338488 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.338699 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.338770 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:09:00.339116 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.335071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:33635 (local address 127.16.61.1:45569)
0619 11:09:00.335377 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.335392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:00.335507 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:00.335890 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:00.335893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:00.335915 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:00.336060 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.336067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.337002 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.337007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.337772 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.337780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.338431 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.338435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.338568 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.338586 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 11:09:00.338676 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 11:09:00.338745 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":71,"threads_started":1}
I20260619 11:09:00.339454 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4732fc01754500a0cb8c0de33dcb06" candidate_uuid: "d8017f1c27b845d39a3da8a4b2146e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" is_pre_election: true
I20260619 11:09:00.339823 3788 raft_consensus.cc:2454] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8017f1c27b845d39a3da8a4b2146e39 in term 0.
I20260619 11:09:00.339802 3654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4732fc01754500a0cb8c0de33dcb06" candidate_uuid: "d8017f1c27b845d39a3da8a4b2146e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467d94bad034ef7932208181e3e0fd2" is_pre_election: true
I20260619 11:09:00.340062 3654 raft_consensus.cc:2454] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8017f1c27b845d39a3da8a4b2146e39 in term 0.
I20260619 11:09:00.340283 3453 leader_election.cc:302] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c39a1bf5b1479d8f5cc10d36e3ae54, d8017f1c27b845d39a3da8a4b2146e39; no voters:
I20260619 11:09:00.340595 3857 raft_consensus.cc:2781] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:00.340682 3857 raft_consensus.cc:482] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:00.340725 3857 raft_consensus.cc:3037] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:00.341706 3857 raft_consensus.cc:504] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.342031 3857 leader_election.cc:288] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [CANDIDATE]: Term 1 election: Requested vote from peers 7467d94bad034ef7932208181e3e0fd2 (127.16.61.2:33635), 02c39a1bf5b1479d8f5cc10d36e3ae54 (127.16.61.3:42213)
I20260619 11:09:00.342391 3788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4732fc01754500a0cb8c0de33dcb06" candidate_uuid: "d8017f1c27b845d39a3da8a4b2146e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54"
I20260619 11:09:00.342418 3654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4732fc01754500a0cb8c0de33dcb06" candidate_uuid: "d8017f1c27b845d39a3da8a4b2146e39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7467d94bad034ef7932208181e3e0fd2"
I20260619 11:09:00.342499 3788 raft_consensus.cc:3037] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:00.342562 3654 raft_consensus.cc:3037] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:00.343506 3788 raft_consensus.cc:2454] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8017f1c27b845d39a3da8a4b2146e39 in term 1.
I20260619 11:09:00.343712 3654 raft_consensus.cc:2454] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8017f1c27b845d39a3da8a4b2146e39 in term 1.
I20260619 11:09:00.343874 3453 leader_election.cc:302] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c39a1bf5b1479d8f5cc10d36e3ae54, d8017f1c27b845d39a3da8a4b2146e39; no voters:
I20260619 11:09:00.344058 3857 raft_consensus.cc:2781] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:00.344290 3857 raft_consensus.cc:686] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [term 1 LEADER]: Becoming Leader. State: Replica: d8017f1c27b845d39a3da8a4b2146e39, State: Running, Role: LEADER
I20260619 11:09:00.344594 3857 consensus_queue.cc:237] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } }
I20260619 11:09:00.347864 3373 catalog_manager.cc:5495] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8017f1c27b845d39a3da8a4b2146e39 (127.16.61.1). New cstate: current_term: 1 leader_uuid: "d8017f1c27b845d39a3da8a4b2146e39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8017f1c27b845d39a3da8a4b2146e39" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 40365 } health_report { overall_health: HEALTHY } } }
I20260619 11:09:00.359506 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.356212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.16.61.1:40365)
0619 11:09:00.356332 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.356335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.356395 (+ 60us) server_negotiation.cc:407] Connection header received
0619 11:09:00.356512 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.356514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.356562 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.356644 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:00.356912 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.357448 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.358321 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.358523 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.358599 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.358843 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:00.359215 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:00.359298 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.359370 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.359415 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 11:09:00.365180 3654 raft_consensus.cc:1261] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 1 FOLLOWER]: Refusing update from remote peer d8017f1c27b845d39a3da8a4b2146e39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:00.365198 3788 raft_consensus.cc:1261] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 1 FOLLOWER]: Refusing update from remote peer d8017f1c27b845d39a3da8a4b2146e39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:00.365882 3862 consensus_queue.cc:1035] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c39a1bf5b1479d8f5cc10d36e3ae54" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:00.366051 3857 consensus_queue.cc:1035] T 3a4732fc01754500a0cb8c0de33dcb06 P d8017f1c27b845d39a3da8a4b2146e39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7467d94bad034ef7932208181e3e0fd2" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:00.373744 3865 mvcc.cc:205] Tried to move back new op lower bound from 7298528626129059840 to 7298528626054037504. Current Snapshot: MvccSnapshot[applied={T|T < 7298528626129059840}]
I20260619 11:09:00.374040 3867 mvcc.cc:205] Tried to move back new op lower bound from 7298528626129059840 to 7298528626054037504. Current Snapshot: MvccSnapshot[applied={T|T < 7298528626129059840}]
I20260619 11:09:00.374742 3866 mvcc.cc:205] Tried to move back new op lower bound from 7298528626129059840 to 7298528626054037504. Current Snapshot: MvccSnapshot[applied={T|T < 7298528626129059840}]
I20260619 11:09:00.389052 3374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:00.389226 3374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:00.391571 3374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6732f2fca34947469d7b35de2e1f5d90: Sending DeleteTablet for 3 replicas of tablet 3a4732fc01754500a0cb8c0de33dcb06
I20260619 11:09:00.392369 3768 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4732fc01754500a0cb8c0de33dcb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:00 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:46768
I20260619 11:09:00.392414 3500 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4732fc01754500a0cb8c0de33dcb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:00 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:41438
I20260619 11:09:00.392455 3634 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4732fc01754500a0cb8c0de33dcb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:00 UTC) from {username='kudu', principal='kudu/127.16.61.62@KRBTEST.COM'} at 127.0.0.1:46286
I20260619 11:09:00.392963 3877 tablet_replica.cc:330] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: stopping tablet replica
I20260619 11:09:00.393276 3877 raft_consensus.cc:2229] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:00.393503 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 3437
I20260619 11:09:00.393605 3877 raft_consensus.cc:2258] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:00.395529 3877 ts_tablet_manager.cc:1905] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:00.395864 3875 tablet_replica.cc:330] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: stopping tablet replica
I20260619 11:09:00.396137 3875 raft_consensus.cc:2229] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:00.396420 3875 raft_consensus.cc:2258] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:00.397590 3875 ts_tablet_manager.cc:1905] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:00.399158 3877 ts_tablet_manager.cc:1918] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:00.399267 3877 log.cc:1196] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/3a4732fc01754500a0cb8c0de33dcb06
I20260619 11:09:00.399607 3877 ts_tablet_manager.cc:1939] T 3a4732fc01754500a0cb8c0de33dcb06 P 7467d94bad034ef7932208181e3e0fd2: Deleting consensus metadata
I20260619 11:09:00.399833 3875 ts_tablet_manager.cc:1918] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:00.399936 3875 log.cc:1196] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/3a4732fc01754500a0cb8c0de33dcb06
I20260619 11:09:00.400219 3875 ts_tablet_manager.cc:1939] T 3a4732fc01754500a0cb8c0de33dcb06 P 02c39a1bf5b1479d8f5cc10d36e3ae54: Deleting consensus metadata
I20260619 11:09:00.400944 3361 catalog_manager.cc:4841] TS 7467d94bad034ef7932208181e3e0fd2 (127.16.61.2:33635): tablet 3a4732fc01754500a0cb8c0de33dcb06 (table test-table [id=b2e1c6f47d654fccb12cc8d6c88010d4]) successfully deleted
I20260619 11:09:00.401196 3359 catalog_manager.cc:4841] TS 02c39a1bf5b1479d8f5cc10d36e3ae54 (127.16.61.3:42213): tablet 3a4732fc01754500a0cb8c0de33dcb06 (table test-table [id=b2e1c6f47d654fccb12cc8d6c88010d4]) successfully deleted
W20260619 11:09:00.401579 3361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:40365 (error 108)
I20260619 11:09:00.401798 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 3571
I20260619 11:09:00.403155 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.402892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:40365 (local address 127.0.0.1:41450)
0619 11:09:00.403015 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.403089 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:40365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 11:09:00.403292 3361 catalog_manager.cc:4584] TS d8017f1c27b845d39a3da8a4b2146e39 (127.16.61.1:40365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a4732fc01754500a0cb8c0de33dcb06: Network error: Client connection negotiation failed: client connection to 127.16.61.1:40365: connect: Connection refused (error 111)
I20260619 11:09:00.407891 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 3705
I20260619 11:09:00.413353 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 3343
2026-06-19T11:09:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2223 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T11:09:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:00Z Disabled control of system clock
I20260619 11:09:00.442752 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:42303
--webserver_interface=127.16.61.62
--webserver_port=0
--builtin_ntp_servers=127.16.61.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:42303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:09:00.548892 3882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:00.549129 3882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:00.549162 3882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:00.553033 3882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:00.553085 3882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:00.553100 3882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:00.553117 3882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:00.553133 3882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:00.557781 3882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:42303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:42303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:00.558692 3882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:00.559983 3882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:00.567953 3887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:00.567975 3888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:00.567962 3890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:00.568154 3882 server_base.cc:1034] running on GCE node
I20260619 11:09:00.568797 3882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:00.569763 3882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:00.570956 3882 hybrid_clock.cc:648] HybridClock initialized: now 1781867340570941 us; error 28 us; skew 500 ppm
I20260619 11:09:00.572747 3882 webserver.cc:460] Webserver started at http://127.16.61.62:44793/ using document root <none> and password file <none>
I20260619 11:09:00.573299 3882 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:00.573391 3882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:00.573614 3882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:00.575551 3882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "09f7a184a3244e23abbd53f6ea14b856"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "cbb417214d9cd7a8aaf96af5a5bd85e9"
server_key_iv: "d767908463b88b1a42ae79bb6979c89f"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.576186 3882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "09f7a184a3244e23abbd53f6ea14b856"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "cbb417214d9cd7a8aaf96af5a5bd85e9"
server_key_iv: "d767908463b88b1a42ae79bb6979c89f"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.579602 3882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:09:00.581857 3896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.582980 3882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 11:09:00.583113 3882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "09f7a184a3244e23abbd53f6ea14b856"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "cbb417214d9cd7a8aaf96af5a5bd85e9"
server_key_iv: "d767908463b88b1a42ae79bb6979c89f"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.583220 3882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:00.605404 3882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:00.606045 3882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:00.606240 3882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:00.613504 3882 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:42303
I20260619 11:09:00.613516 3948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:42303 every 8 connection(s)
I20260619 11:09:00.614391 3882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:00.617340 3949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:00.617899 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 3882
I20260619 11:09:00.618043 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:00.618327 16628 external_mini_cluster.cc:1428] Setting key e19e3d0b67b6fd8280d340df8f97afc3
I20260619 11:09:00.623899 3949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856: Bootstrap starting.
I20260619 11:09:00.626293 3949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:00.626995 3949 log.cc:824] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:00.628074 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.619725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.16.61.62:42303)
0619 11:09:00.620375 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.620396 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.620440 (+ 44us) server_negotiation.cc:407] Connection header received
0619 11:09:00.621275 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.621299 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.621780 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.622130 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:00.622559 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.624001 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.624865 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.625526 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.625686 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.626459 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:00.626464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:00.626476 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:00.626835 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:00.626887 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:00.626906 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:00.627013 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.627138 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.627372 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":91,"threads_started":1}
I20260619 11:09:00.628885 3949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856: No bootstrap required, opened a new log
I20260619 11:09:00.632381 3949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a184a3244e23abbd53f6ea14b856" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42303 } }
I20260619 11:09:00.632608 3949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:00.632687 3949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f7a184a3244e23abbd53f6ea14b856, State: Initialized, Role: FOLLOWER
I20260619 11:09:00.633126 3949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a184a3244e23abbd53f6ea14b856" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42303 } }
I20260619 11:09:00.633304 3954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:00.633447 3954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a184a3244e23abbd53f6ea14b856" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42303 } }
I20260619 11:09:00.633276 3949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:00.633790 3954 leader_election.cc:302] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f7a184a3244e23abbd53f6ea14b856; no voters:
I20260619 11:09:00.633944 3949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:00.634035 3949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:00.634282 3954 leader_election.cc:288] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260619 11:09:00.634953 3949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a184a3244e23abbd53f6ea14b856" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42303 } }
I20260619 11:09:00.635114 3949 leader_election.cc:302] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f7a184a3244e23abbd53f6ea14b856; no voters:
I20260619 11:09:00.635167 3955 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 11:09:00.635223 3949 leader_election.cc:288] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:00.635363 3954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:00.635453 3954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [term 1 LEADER]: Becoming Leader. State: Replica: 09f7a184a3244e23abbd53f6ea14b856, State: Running, Role: LEADER
I20260619 11:09:00.635777 3954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a184a3244e23abbd53f6ea14b856" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42303 } }
I20260619 11:09:00.636310 3949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:00.636806 3954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f7a184a3244e23abbd53f6ea14b856. Latest consensus state: current_term: 1 leader_uuid: "09f7a184a3244e23abbd53f6ea14b856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a184a3244e23abbd53f6ea14b856" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42303 } } }
I20260619 11:09:00.636914 3954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:00.637318 3955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f7a184a3244e23abbd53f6ea14b856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a184a3244e23abbd53f6ea14b856" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 42303 } } }
I20260619 11:09:00.637434 3955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:00.637456 3959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:00.640569 3959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:00.646090 3959 catalog_manager.cc:1347] Generated new cluster ID: 61d9e968b15a4c0e936d48ee8edbf925
I20260619 11:09:00.646152 3959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:00.653367 3959 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:00.654191 3959 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:00.664633 3959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856: Generated new TSK 0
I20260619 11:09:00.665387 3959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:00.674806 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.1:0
--local_ip_for_outbound_sockets=127.16.61.1
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42303
--builtin_ntp_servers=127.16.61.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 11:09:00.781643 3973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:00.781905 3973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:00.781942 3973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:00.785859 3973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:00.785905 3973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:00.785956 3973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.1
I20260619 11:09:00.790632 3973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.61.1
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.1
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:00.791596 3973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:00.792900 3973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:00.801973 3979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:00.801973 3978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:00.801980 3981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:00.802191 3973 server_base.cc:1034] running on GCE node
I20260619 11:09:00.802609 3973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:00.803241 3973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:00.804402 3973 hybrid_clock.cc:648] HybridClock initialized: now 1781867340804389 us; error 24 us; skew 500 ppm
I20260619 11:09:00.806209 3973 webserver.cc:460] Webserver started at http://127.16.61.1:41031/ using document root <none> and password file <none>
I20260619 11:09:00.806757 3973 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:00.806819 3973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:00.807062 3973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:00.809027 3973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/data/instance:
uuid: "24a20bec58ea416e8c8ada2f51f16ed1"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "b39c3d070b275ef05470c792b7df7528"
server_key_iv: "17ae1243cbd4f289c6111fc61dedc6e6"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.809562 3973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance:
uuid: "24a20bec58ea416e8c8ada2f51f16ed1"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "b39c3d070b275ef05470c792b7df7528"
server_key_iv: "17ae1243cbd4f289c6111fc61dedc6e6"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.813033 3973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:00.815285 3987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.816444 3973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:00.816533 3973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/wal
uuid: "24a20bec58ea416e8c8ada2f51f16ed1"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "b39c3d070b275ef05470c792b7df7528"
server_key_iv: "17ae1243cbd4f289c6111fc61dedc6e6"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.816610 3973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:00.827309 3973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:00.827842 3973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:00.827971 3973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:00.828459 3973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:00.829236 3973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:00.829280 3973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.829316 3973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:00.829332 3973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.838074 3973 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.1:34971
I20260619 11:09:00.838078 4100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.1:34971 every 8 connection(s)
I20260619 11:09:00.838961 3973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/data/info.pb
I20260619 11:09:00.840473 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 3973
I20260619 11:09:00.840624 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-0/wal/instance
I20260619 11:09:00.840903 16628 external_mini_cluster.cc:1428] Setting key 99b6172d210d74da7e5aedb89df55f02
I20260619 11:09:00.843000 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.2:0
--local_ip_for_outbound_sockets=127.16.61.2
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42303
--builtin_ntp_servers=127.16.61.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:09:00.849032 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.840662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.1:60803 (local address 127.16.61.62:42303)
0619 11:09:00.840784 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:00.840788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:00.841878 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 11:09:00.843579 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:00.843582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:00.843626 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:00.843762 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:00.844804 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.845644 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.846826 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.847254 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.847382 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.848448 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:00.848455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:00.848460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:00.848477 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:00.848503 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:00.848517 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:00.848599 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:00.848755 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 11:09:00.848832 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:09:00.849669 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:00.841112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42303 (local address 127.16.61.1:60803)
0619 11:09:00.841741 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:00.841777 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:00.843359 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:00.843927 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:00.843939 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:00.844044 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:00.844658 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.844672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.845784 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.845786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.846668 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:00.846682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:00.847394 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:00.847397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:00.847525 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:00.848197 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:09:00.848198 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:00.848235 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:09:00.848290 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:00.848613 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:00.848619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:00.848623 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:09:00.848712 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 11:09:00.848907 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":146,"threads_started":1}
I20260619 11:09:00.850909 4101 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42303
I20260619 11:09:00.851166 4101 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:00.851629 4101 heartbeater.cc:511] Master 127.16.61.62:42303 requested a full tablet report, sending...
I20260619 11:09:00.853276 3913 ts_manager.cc:194] Registered new tserver with Master: 24a20bec58ea416e8c8ada2f51f16ed1 (127.16.61.1:34971)
I20260619 11:09:00.854879 3913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.1:60803
W20260619 11:09:00.950627 4104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:00.950896 4104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:00.950963 4104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:00.955626 4104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:00.955713 4104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:00.955812 4104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.2
I20260619 11:09:00.960494 4104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.61.2
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.2
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:00.961490 4104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:00.962769 4104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:00.971563 4112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:00.971557 4109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:00.971560 4110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:00.971797 4104 server_base.cc:1034] running on GCE node
I20260619 11:09:00.972332 4104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:00.972925 4104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:00.974093 4104 hybrid_clock.cc:648] HybridClock initialized: now 1781867340974069 us; error 34 us; skew 500 ppm
I20260619 11:09:00.975890 4104 webserver.cc:460] Webserver started at http://127.16.61.2:45805/ using document root <none> and password file <none>
I20260619 11:09:00.976418 4104 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:00.976476 4104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:00.976699 4104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:00.978562 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/data/instance:
uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "a63a28e2b3f5c54263214e88d6fa14fa"
server_key_iv: "e694a142abcac8ee455a8e95d778c630"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.979094 4104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance:
uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "a63a28e2b3f5c54263214e88d6fa14fa"
server_key_iv: "e694a142abcac8ee455a8e95d778c630"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.982415 4104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 11:09:00.984640 4118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:00.985754 4104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:00.985867 4104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/wal
uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979"
format_stamp: "Formatted at 2026-06-19 11:09:00 on dist-test-slave-8vcz"
server_key: "a63a28e2b3f5c54263214e88d6fa14fa"
server_key_iv: "e694a142abcac8ee455a8e95d778c630"
server_key_version: "encryptionkey@0"
I20260619 11:09:00.985970 4104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:01.025550 4104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:01.026191 4104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:01.026338 4104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:01.026856 4104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:01.027738 4104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:01.027783 4104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:01.027822 4104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:01.027845 4104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:01.036621 4104 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.2:45683
I20260619 11:09:01.036731 4231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.2:45683 every 8 connection(s)
I20260619 11:09:01.037729 4104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/data/info.pb
I20260619 11:09:01.039417 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 4104
I20260619 11:09:01.039551 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/wal/instance
I20260619 11:09:01.039894 16628 external_mini_cluster.cc:1428] Setting key 8c1002c899dfef68490b64a2fcd03ed0
I20260619 11:09:01.042033 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.61.3:0
--local_ip_for_outbound_sockets=127.16.61.3
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42303
--builtin_ntp_servers=127.16.61.20:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 11:09:01.048266 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.039616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:39877 (local address 127.16.61.62:42303)
0619 11:09:01.039785 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.039790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.040954 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 11:09:01.042765 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.042768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.042811 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.042878 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:01.043998 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.044832 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.046034 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.046470 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.046602 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.047653 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:01.047658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:01.047661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:01.047709 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:01.047735 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:01.047750 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:01.047835 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.047976 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.048050 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 11:09:01.048959 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.040099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42303 (local address 127.16.61.2:39877)
0619 11:09:01.040820 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:01.040859 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:01.042543 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:01.043046 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:01.043057 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:01.043165 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:01.043857 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.043873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.044964 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.044967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.045910 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.045919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.046617 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.046619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.046755 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.047391 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:09:01.047393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:01.047431 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:09:01.047499 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:01.047832 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:01.047839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:01.047843 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 11:09:01.047939 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 11:09:01.048156 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":145,"threads_started":1}
I20260619 11:09:01.050163 4232 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42303
I20260619 11:09:01.050415 4232 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:01.050907 4232 heartbeater.cc:511] Master 127.16.61.62:42303 requested a full tablet report, sending...
I20260619 11:09:01.051972 3913 ts_manager.cc:194] Registered new tserver with Master: a3bc0ba6b6a44e10be1b8ef0b011f979 (127.16.61.2:45683)
I20260619 11:09:01.052542 3913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.2:39877
W20260619 11:09:01.150086 4235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:01.150321 4235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:01.150357 4235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:01.155411 4235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:01.155463 4235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:01.155547 4235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.61.3
I20260619 11:09:01.160195 4235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.61.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.61.3
--webserver_port=0
--tserver_master_addrs=127.16.61.62:42303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.61.3
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:01.161128 4235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:01.162417 4235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:01.171761 4243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:01.171885 4240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:01.171813 4241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:01.172066 4235 server_base.cc:1034] running on GCE node
I20260619 11:09:01.172484 4235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:01.173054 4235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:01.174233 4235 hybrid_clock.cc:648] HybridClock initialized: now 1781867341174222 us; error 24 us; skew 500 ppm
I20260619 11:09:01.175978 4235 webserver.cc:460] Webserver started at http://127.16.61.3:46841/ using document root <none> and password file <none>
I20260619 11:09:01.176532 4235 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:01.176620 4235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:01.176839 4235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:01.178717 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/data/instance:
uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e"
format_stamp: "Formatted at 2026-06-19 11:09:01 on dist-test-slave-8vcz"
server_key: "5b21c0d2f4e5109810bd383c01e8d9db"
server_key_iv: "2ae7a86d3d743deedaaa01855359201a"
server_key_version: "encryptionkey@0"
I20260619 11:09:01.179251 4235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance:
uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e"
format_stamp: "Formatted at 2026-06-19 11:09:01 on dist-test-slave-8vcz"
server_key: "5b21c0d2f4e5109810bd383c01e8d9db"
server_key_iv: "2ae7a86d3d743deedaaa01855359201a"
server_key_version: "encryptionkey@0"
I20260619 11:09:01.182638 4235 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:01.184880 4249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:01.185948 4235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:01.186074 4235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/wal
uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e"
format_stamp: "Formatted at 2026-06-19 11:09:01 on dist-test-slave-8vcz"
server_key: "5b21c0d2f4e5109810bd383c01e8d9db"
server_key_iv: "2ae7a86d3d743deedaaa01855359201a"
server_key_version: "encryptionkey@0"
I20260619 11:09:01.186179 4235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:01.197723 4235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:01.198307 4235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:01.198518 4235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:01.199054 4235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 11:09:01.199931 4235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 11:09:01.199980 4235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:01.200017 4235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 11:09:01.200032 4235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:01.208756 4235 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.3:44035
I20260619 11:09:01.208791 4362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.3:44035 every 8 connection(s)
I20260619 11:09:01.209580 4235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/data/info.pb
I20260619 11:09:01.218286 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.211194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.3:39745 (local address 127.16.61.62:42303)
0619 11:09:01.211336 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.211340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.212392 (+ 1052us) server_negotiation.cc:407] Connection header received
0619 11:09:01.213743 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.213746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.213804 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.213912 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:01.214760 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.215288 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.216405 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.216631 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.216749 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.217794 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:01.217798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:01.217801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:01.217813 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:01.217841 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:01.217849 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:01.217925 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.218081 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.218142 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:09:01.218467 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 4235
I20260619 11:09:01.218577 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/wal/instance
I20260619 11:09:01.218850 16628 external_mini_cluster.cc:1428] Setting key 710beaf8decf3ab23a9712162bc2f3f1
I20260619 11:09:01.219225 4365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.211570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:42303 (local address 127.16.61.3:39745)
0619 11:09:01.212239 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:01.212277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:01.213538 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:01.214077 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:01.214083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:01.214168 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:01.214612 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.214625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.215413 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.215417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.216278 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.216287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.216782 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.216787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.216943 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.217548 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 11:09:01.217549 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:01.217585 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 11:09:01.217642 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:01.217937 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:01.217947 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:01.217954 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 11:09:01.218080 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 11:09:01.218364 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260619 11:09:01.220278 4363 heartbeater.cc:347] Connected to a master server at 127.16.61.62:42303
I20260619 11:09:01.220501 4363 heartbeater.cc:464] Registering TS with master...
I20260619 11:09:01.221043 4363 heartbeater.cc:511] Master 127.16.61.62:42303 requested a full tablet report, sending...
I20260619 11:09:01.221951 3913 ts_manager.cc:194] Registered new tserver with Master: 8ffa2cc501eb482aaf5f9c913dd2e54e (127.16.61.3:44035)
I20260619 11:09:01.222506 3913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.61.3:39745
I20260619 11:09:01.232641 16628 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 11:09:01.237991 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.234512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.16.61.62:42303)
0619 11:09:01.234629 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.234632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.234788 (+ 156us) server_negotiation.cc:407] Connection header received
0619 11:09:01.234884 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.234886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.234921 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.234984 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:01.235216 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.235705 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.236531 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.236760 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.236879 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.237544 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:01.237547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:01.237549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:01.237559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:01.237575 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:01.237583 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:01.237671 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.237798 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.237842 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:09:01.240518 3913 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 11:09:01.243327 3913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 11:09:01.257365 4378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.252003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.2:45683 (local address 127.0.0.1:48346)
0619 11:09:01.252704 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:01.252716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:01.252790 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:01.253571 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:01.253573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:01.253590 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:01.253672 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.253677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.254987 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.254990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.256064 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.256070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.256825 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.256828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.256929 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.256965 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 11:09:01.257079 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 11:09:01.257201 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260619 11:09:01.257373 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.252381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.16.61.2:45683)
0619 11:09:01.253032 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.253044 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.253064 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:09:01.253135 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.253139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.253288 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.253428 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:01.253794 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.254862 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.256170 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.256688 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.256823 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.256995 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.257107 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.257201 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":89,"threads_started":1}
I20260619 11:09:01.257637 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.251413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:34971 (local address 127.0.0.1:58382)
0619 11:09:01.252482 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:01.252494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:01.252572 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:01.253174 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:01.253177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:01.253194 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:01.253296 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.253302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.254411 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.254415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.255798 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.255808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.257053 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.257057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.257313 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.257345 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 11:09:01.257415 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 11:09:01.257495 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":119,"thread_start_us":82,"threads_started":1}
I20260619 11:09:01.258003 4375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.251317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:44035 (local address 127.0.0.1:35426)
0619 11:09:01.251967 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:01.252015 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:01.252140 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:01.252889 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:01.252897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:01.252919 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:01.253050 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.253059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.254306 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.254309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.255808 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.255817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.257053 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.257057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.257284 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.257344 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 11:09:01.257415 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 11:09:01.257868 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":406,"thread_start_us":68,"threads_started":1}
I20260619 11:09:01.258245 4377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.251940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.16.61.1:34971)
0619 11:09:01.252359 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.252370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.252657 (+ 287us) server_negotiation.cc:407] Connection header received
0619 11:09:01.252731 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.252737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.252900 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.253029 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:01.253418 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.254299 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.255920 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.256706 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.256813 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.256988 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.257963 (+ 975us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.258070 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":114,"threads_started":1}
I20260619 11:09:01.258649 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.251416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.16.61.3:44035)
0619 11:09:01.252397 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.252409 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.252430 (+ 21us) server_negotiation.cc:407] Connection header received
0619 11:09:01.252505 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.252515 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.252719 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.252874 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:01.253174 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.254178 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.255970 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.256705 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.256818 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.256989 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.258392 (+ 1403us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.258494 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":106,"threads_started":1}
I20260619 11:09:01.260100 4166 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6a188e79248a4baf211a58b324607 (DEFAULT_TABLE table=test-table [id=dec5e3b8caaf4d70a3e8eb4a68c01d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:01.260423 4035 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6a188e79248a4baf211a58b324607 (DEFAULT_TABLE table=test-table [id=dec5e3b8caaf4d70a3e8eb4a68c01d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:01.261059 4166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa6a188e79248a4baf211a58b324607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:01.261317 4035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa6a188e79248a4baf211a58b324607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:01.266853 4380 tablet_bootstrap.cc:492] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1: Bootstrap starting.
I20260619 11:09:01.268121 4381 tablet_bootstrap.cc:492] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Bootstrap starting.
I20260619 11:09:01.269090 4380 tablet_bootstrap.cc:654] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:01.269778 4381 tablet_bootstrap.cc:654] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:01.269932 4380 log.cc:824] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:01.270551 4381 log.cc:824] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:01.270460 4297 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa6a188e79248a4baf211a58b324607 (DEFAULT_TABLE table=test-table [id=dec5e3b8caaf4d70a3e8eb4a68c01d86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 11:09:01.271481 4297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa6a188e79248a4baf211a58b324607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:01.272101 4380 tablet_bootstrap.cc:492] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1: No bootstrap required, opened a new log
I20260619 11:09:01.272236 4381 tablet_bootstrap.cc:492] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: No bootstrap required, opened a new log
I20260619 11:09:01.272293 4380 ts_tablet_manager.cc:1397] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 11:09:01.272413 4381 ts_tablet_manager.cc:1397] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260619 11:09:01.275413 4381 raft_consensus.cc:348] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.275636 4381 raft_consensus.cc:374] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:01.275702 4381 raft_consensus.cc:729] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bc0ba6b6a44e10be1b8ef0b011f979, State: Initialized, Role: FOLLOWER
I20260619 11:09:01.276111 4381 consensus_queue.cc:260] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.276762 4232 heartbeater.cc:503] Master 127.16.61.62:42303 was elected leader, sending a full tablet report...
I20260619 11:09:01.277031 4381 ts_tablet_manager.cc:1428] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 11:09:01.276424 4380 raft_consensus.cc:348] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.277295 4380 raft_consensus.cc:374] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:01.277379 4380 raft_consensus.cc:729] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a20bec58ea416e8c8ada2f51f16ed1, State: Initialized, Role: FOLLOWER
I20260619 11:09:01.277781 4380 consensus_queue.cc:260] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.278396 4101 heartbeater.cc:503] Master 127.16.61.62:42303 was elected leader, sending a full tablet report...
I20260619 11:09:01.278683 4380 ts_tablet_manager.cc:1428] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260619 11:09:01.279888 4386 tablet_bootstrap.cc:492] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Bootstrap starting.
I20260619 11:09:01.281865 4386 tablet_bootstrap.cc:654] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:01.282509 4386 log.cc:824] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:01.284099 4386 tablet_bootstrap.cc:492] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: No bootstrap required, opened a new log
I20260619 11:09:01.284318 4386 ts_tablet_manager.cc:1397] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 11:09:01.287173 4386 raft_consensus.cc:348] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.287387 4386 raft_consensus.cc:374] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:01.287449 4386 raft_consensus.cc:729] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffa2cc501eb482aaf5f9c913dd2e54e, State: Initialized, Role: FOLLOWER
I20260619 11:09:01.287863 4386 consensus_queue.cc:260] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.288465 4363 heartbeater.cc:503] Master 127.16.61.62:42303 was elected leader, sending a full tablet report...
W20260619 11:09:01.288694 4233 tablet.cc:2369] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:01.288740 4386 ts_tablet_manager.cc:1428] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 11:09:01.340107 4102 tablet.cc:2369] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:01.423828 4384 raft_consensus.cc:482] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 11:09:01.424041 4384 raft_consensus.cc:504] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.425253 4384 leader_election.cc:288] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ffa2cc501eb482aaf5f9c913dd2e54e (127.16.61.3:44035), 24a20bec58ea416e8c8ada2f51f16ed1 (127.16.61.1:34971)
I20260619 11:09:01.429347 4377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.425651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:49297 (local address 127.16.61.1:34971)
0619 11:09:01.425783 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.425787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.426166 (+ 379us) server_negotiation.cc:407] Connection header received
0619 11:09:01.426293 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.426296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.426343 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.426424 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:01.426708 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.427243 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.428357 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.428724 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.428840 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.428899 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.429139 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.429213 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:09:01.429356 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.425438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.3:44035 (local address 127.16.61.2:60725)
0619 11:09:01.425826 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:01.425841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:01.425971 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:01.426246 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:01.426251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:01.426271 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:01.426403 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.426414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.427209 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.427212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.428186 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.428195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.428843 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.428847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.428976 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.429026 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 11:09:01.429117 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 11:09:01.429216 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":64}
I20260619 11:09:01.429358 4376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.425561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.61.2:60725 (local address 127.16.61.3:44035)
0619 11:09:01.425699 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.425706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.425951 (+ 245us) server_negotiation.cc:407] Connection header received
0619 11:09:01.426064 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.426067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.426132 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.426214 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 11:09:01.426521 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.427100 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.428306 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.428723 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.428841 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.428899 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.429133 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.429207 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:09:01.429384 4389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.425551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:34971 (local address 127.16.61.2:49297)
0619 11:09:01.426066 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:01.426079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:01.426191 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:01.426443 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:01.426445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:01.426477 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 11:09:01.426599 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.426605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.427358 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.427361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.428215 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:01.428224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.428845 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.428849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:01.428994 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.429028 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 11:09:01.429116 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 11:09:01.429221 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":100,"thread_start_us":182,"threads_started":1}
I20260619 11:09:01.430053 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa6a188e79248a4baf211a58b324607" candidate_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" is_pre_election: true
I20260619 11:09:01.430047 4055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa6a188e79248a4baf211a58b324607" candidate_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" is_pre_election: true
I20260619 11:09:01.430341 4317 raft_consensus.cc:2454] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3bc0ba6b6a44e10be1b8ef0b011f979 in term 0.
I20260619 11:09:01.430419 4055 raft_consensus.cc:2454] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3bc0ba6b6a44e10be1b8ef0b011f979 in term 0.
I20260619 11:09:01.430830 4119 leader_election.cc:302] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ffa2cc501eb482aaf5f9c913dd2e54e, a3bc0ba6b6a44e10be1b8ef0b011f979; no voters:
I20260619 11:09:01.431128 4384 raft_consensus.cc:2781] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 11:09:01.431236 4384 raft_consensus.cc:482] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 11:09:01.431298 4384 raft_consensus.cc:3037] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:01.432476 4384 raft_consensus.cc:504] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.432758 4384 leader_election.cc:288] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [CANDIDATE]: Term 1 election: Requested vote from peers 8ffa2cc501eb482aaf5f9c913dd2e54e (127.16.61.3:44035), 24a20bec58ea416e8c8ada2f51f16ed1 (127.16.61.1:34971)
I20260619 11:09:01.433162 4055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa6a188e79248a4baf211a58b324607" candidate_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a20bec58ea416e8c8ada2f51f16ed1"
I20260619 11:09:01.433173 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa6a188e79248a4baf211a58b324607" candidate_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e"
I20260619 11:09:01.433282 4317 raft_consensus.cc:3037] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:01.433300 4055 raft_consensus.cc:3037] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:01.434489 4317 raft_consensus.cc:2454] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3bc0ba6b6a44e10be1b8ef0b011f979 in term 1.
I20260619 11:09:01.434489 4055 raft_consensus.cc:2454] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3bc0ba6b6a44e10be1b8ef0b011f979 in term 1.
I20260619 11:09:01.434780 4122 leader_election.cc:302] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a20bec58ea416e8c8ada2f51f16ed1, a3bc0ba6b6a44e10be1b8ef0b011f979; no voters:
I20260619 11:09:01.434927 4384 raft_consensus.cc:2781] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:01.435141 4384 raft_consensus.cc:686] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 1 LEADER]: Becoming Leader. State: Replica: a3bc0ba6b6a44e10be1b8ef0b011f979, State: Running, Role: LEADER
I20260619 11:09:01.435454 4384 consensus_queue.cc:237] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } }
I20260619 11:09:01.438829 3912 catalog_manager.cc:5495] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3bc0ba6b6a44e10be1b8ef0b011f979 (127.16.61.2). New cstate: current_term: 1 leader_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bc0ba6b6a44e10be1b8ef0b011f979" member_type: VOTER last_known_addr { host: "127.16.61.2" port: 45683 } health_report { overall_health: HEALTHY } } }
W20260619 11:09:01.460407 4364 tablet.cc:2369] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:01.490525 4379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.486895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.16.61.2:45683)
0619 11:09:01.487048 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:01.487057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:01.487087 (+ 30us) server_negotiation.cc:407] Connection header received
0619 11:09:01.487155 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:01.487157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:01.487208 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:01.487319 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:01.487580 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.488137 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.489002 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:01.489236 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:01.489355 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:01.489620 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:01.490020 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:01.490149 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:01.490282 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 11:09:01.490326 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 11:09:01.496107 4317 raft_consensus.cc:1261] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 1 FOLLOWER]: Refusing update from remote peer a3bc0ba6b6a44e10be1b8ef0b011f979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:01.496109 4055 raft_consensus.cc:1261] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 1 FOLLOWER]: Refusing update from remote peer a3bc0ba6b6a44e10be1b8ef0b011f979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:01.496766 4384 consensus_queue.cc:1035] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a20bec58ea416e8c8ada2f51f16ed1" member_type: VOTER last_known_addr { host: "127.16.61.1" port: 34971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:01.496925 4390 consensus_queue.cc:1035] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ffa2cc501eb482aaf5f9c913dd2e54e" member_type: VOTER last_known_addr { host: "127.16.61.3" port: 44035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:01.503834 4394 mvcc.cc:205] Tried to move back new op lower bound from 7298528630760906752 to 7298528630522400768. Current Snapshot: MvccSnapshot[applied={T|T < 7298528630760906752}]
I20260619 11:09:01.512385 3913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:01.512542 3913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 11:09:01.514945 3913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09f7a184a3244e23abbd53f6ea14b856: Sending DeleteTablet for 3 replicas of tablet 9fa6a188e79248a4baf211a58b324607
I20260619 11:09:01.515615 4297 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa6a188e79248a4baf211a58b324607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:01 UTC) from {username='slave'} at 127.0.0.1:35426
I20260619 11:09:01.515707 4166 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa6a188e79248a4baf211a58b324607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:01 UTC) from {username='slave'} at 127.0.0.1:48346
I20260619 11:09:01.515709 4035 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa6a188e79248a4baf211a58b324607 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 11:09:01 UTC) from {username='slave'} at 127.0.0.1:58382
I20260619 11:09:01.516145 4404 tablet_replica.cc:330] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1: stopping tablet replica
I20260619 11:09:01.516471 4402 tablet_replica.cc:330] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: stopping tablet replica
I20260619 11:09:01.516471 4404 raft_consensus.cc:2229] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:01.516734 4402 raft_consensus.cc:2229] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 11:09:01.516767 4404 raft_consensus.cc:2258] T 9fa6a188e79248a4baf211a58b324607 P 24a20bec58ea416e8c8ada2f51f16ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:01.516871 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 3973
I20260619 11:09:01.517104 4403 tablet_replica.cc:330] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: stopping tablet replica
I20260619 11:09:01.517355 4403 raft_consensus.cc:2229] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 1 LEADER]: Raft consensus shutting down.
I20260619 11:09:01.517704 4403 raft_consensus.cc:2258] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:01.518040 4402 raft_consensus.cc:2258] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 11:09:01.519126 4403 ts_tablet_manager.cc:1905] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:01.519388 4402 ts_tablet_manager.cc:1905] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 11:09:01.521940 4402 ts_tablet_manager.cc:1918] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:01.522028 4402 log.cc:1196] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-2/wal/wals/9fa6a188e79248a4baf211a58b324607
I20260619 11:09:01.522325 4402 ts_tablet_manager.cc:1939] T 9fa6a188e79248a4baf211a58b324607 P 8ffa2cc501eb482aaf5f9c913dd2e54e: Deleting consensus metadata
I20260619 11:09:01.522491 4403 ts_tablet_manager.cc:1918] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 11:09:01.522574 4403 log.cc:1196] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Deleting WAL directory at /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781867094678158-16628-0/minicluster-data/ts-1/wal/wals/9fa6a188e79248a4baf211a58b324607
I20260619 11:09:01.522868 4403 ts_tablet_manager.cc:1939] T 9fa6a188e79248a4baf211a58b324607 P a3bc0ba6b6a44e10be1b8ef0b011f979: Deleting consensus metadata
I20260619 11:09:01.523406 3897 catalog_manager.cc:4841] TS 8ffa2cc501eb482aaf5f9c913dd2e54e (127.16.61.3:44035): tablet 9fa6a188e79248a4baf211a58b324607 (table test-table [id=dec5e3b8caaf4d70a3e8eb4a68c01d86]) successfully deleted
I20260619 11:09:01.523931 3900 catalog_manager.cc:4841] TS a3bc0ba6b6a44e10be1b8ef0b011f979 (127.16.61.2:45683): tablet 9fa6a188e79248a4baf211a58b324607 (table test-table [id=dec5e3b8caaf4d70a3e8eb4a68c01d86]) successfully deleted
W20260619 11:09:01.525671 3900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.61.1:34971 (error 108)
I20260619 11:09:01.526465 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 4104
I20260619 11:09:01.526708 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:01.526406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.1:34971 (local address 127.0.0.1:35938)
0619 11:09:01.526539 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:01.526615 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.1:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 11:09:01.527432 3900 catalog_manager.cc:4584] TS 24a20bec58ea416e8c8ada2f51f16ed1 (127.16.61.1:34971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fa6a188e79248a4baf211a58b324607: Network error: Client connection negotiation failed: client connection to 127.16.61.1:34971: connect: Connection refused (error 111)
I20260619 11:09:01.532119 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 4235
I20260619 11:09:01.539283 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 3882
2026-06-19T11:09:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1124 ms)
[----------] 6 tests from EncryptionPolicyTest (9869 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:01 dist-test-slave-8vcz krb5kdc[4408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:01 dist-test-slave-8vcz krb5kdc[4408](info): set up 2 sockets
Jun 19 11:09:01 dist-test-slave-8vcz krb5kdc[4408](info): commencing operation
krb5kdc: starting...
W20260619 11:09:02.567786 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:02 dist-test-slave-8vcz krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:02.684469 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:02.788859 4421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:02.789090 4421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:02.789129 4421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:02.793066 4421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:02.793119 4421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:02.793138 4421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:02.793154 4421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:02.793169 4421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:02.798149 4421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:02.799113 4421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:02.800096 4421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:02.808012 4427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:02.808025 4426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:02.808118 4429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:02.808192 4421 server_base.cc:1034] running on GCE node
I20260619 11:09:02.808898 4421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:02.809957 4421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:02.811151 4421 hybrid_clock.cc:648] HybridClock initialized: now 1781867342811123 us; error 38 us; skew 500 ppm
Jun 19 11:09:02 dist-test-slave-8vcz krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:02.817171 4421 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:02.818095 4421 webserver.cc:460] Webserver started at http://127.0.0.1:34341/ using document root <none> and password file <none>
I20260619 11:09:02.818614 4421 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:02.818673 4421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:02.818900 4421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:02.820863 4421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "3fa47ecbd6014603bb76d7d2cb337e7f"
format_stamp: "Formatted at 2026-06-19 11:09:02 on dist-test-slave-8vcz"
server_key: "7c9fb84abfb36ad6d48bc719933bea2a"
server_key_iv: "f61982d7f14abea316aefb0b600c3054"
server_key_version: "encryptionkey@0"
I20260619 11:09:02.821409 4421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "3fa47ecbd6014603bb76d7d2cb337e7f"
format_stamp: "Formatted at 2026-06-19 11:09:02 on dist-test-slave-8vcz"
server_key: "7c9fb84abfb36ad6d48bc719933bea2a"
server_key_iv: "f61982d7f14abea316aefb0b600c3054"
server_key_version: "encryptionkey@0"
I20260619 11:09:02.824854 4421 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 11:09:02.827085 4436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:02.828249 4421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:02.828382 4421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "3fa47ecbd6014603bb76d7d2cb337e7f"
format_stamp: "Formatted at 2026-06-19 11:09:02 on dist-test-slave-8vcz"
server_key: "7c9fb84abfb36ad6d48bc719933bea2a"
server_key_iv: "f61982d7f14abea316aefb0b600c3054"
server_key_version: "encryptionkey@0"
I20260619 11:09:02.828483 4421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:02.839923 4421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:02.842463 4421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:02.842651 4421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:02.849296 4421 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40235
I20260619 11:09:02.849381 4488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40235 every 8 connection(s)
I20260619 11:09:02.850149 4421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:02.852596 4489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:02.857513 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f: Bootstrap starting.
I20260619 11:09:02.859769 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 4421
I20260619 11:09:02.859851 4489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:02.859903 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:02.860193 16628 external_mini_cluster.cc:1428] Setting key 56b59260959940fcfea1ed33b911c000
I20260619 11:09:02.860630 4489 log.cc:824] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:02.862754 4489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f: No bootstrap required, opened a new log
I20260619 11:09:02.866894 4489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa47ecbd6014603bb76d7d2cb337e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260619 11:09:02.867148 4489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:02.867218 4489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa47ecbd6014603bb76d7d2cb337e7f, State: Initialized, Role: FOLLOWER
I20260619 11:09:02.867645 4489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa47ecbd6014603bb76d7d2cb337e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260619 11:09:02.867848 4489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:02.867937 4489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:02.868053 4489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:02.868909 4489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa47ecbd6014603bb76d7d2cb337e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260619 11:09:02.869273 4489 leader_election.cc:302] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fa47ecbd6014603bb76d7d2cb337e7f; no voters:
Jun 19 11:09:02 dist-test-slave-8vcz krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:09:02.869603 4489 leader_election.cc:288] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:02.869659 4494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:02.869843 4494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [term 1 LEADER]: Becoming Leader. State: Replica: 3fa47ecbd6014603bb76d7d2cb337e7f, State: Running, Role: LEADER
I20260619 11:09:02.870141 4494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa47ecbd6014603bb76d7d2cb337e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } }
I20260619 11:09:02.870800 4489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:02.871277 4496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fa47ecbd6014603bb76d7d2cb337e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa47ecbd6014603bb76d7d2cb337e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } } }
I20260619 11:09:02.871415 4496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:02.872098 4495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fa47ecbd6014603bb76d7d2cb337e7f. Latest consensus state: current_term: 1 leader_uuid: "3fa47ecbd6014603bb76d7d2cb337e7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fa47ecbd6014603bb76d7d2cb337e7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40235 } } }
I20260619 11:09:02.872222 4495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f [sys.catalog]: This master's current role is: LEADER
W20260619 11:09:02.875831 4509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:02.875900 4509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:02.875972 4499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:02.876361 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:02.861261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60015 (local address 127.0.0.1:40235)
0619 11:09:02.861684 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:02.861702 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:02.861742 (+ 40us) server_negotiation.cc:407] Connection header received
0619 11:09:02.862547 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:02.862568 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:02.862969 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:02.863311 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:02.864034 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:02.865511 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:02.866392 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:02.867058 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:02.867239 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:02.870243 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:02.870251 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:02.870267 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:02.870301 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:02.872966 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:02.873568 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:02.873573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:02.873579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:02.873710 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:02.874186 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:02.874189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:02.874191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:02.874653 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:02.874870 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:02.875190 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 11:09:02.875425 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260619 11:09:02.876765 4499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:02.879503 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:02.882203 4499 catalog_manager.cc:1347] Generated new cluster ID: d62a02caf1fb427a99cfd3e41db8affa
I20260619 11:09:02.882266 4499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:02.890658 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:02.881691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51943 (local address 127.0.0.1:40235)
0619 11:09:02.881788 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:02.881792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:02.882301 (+ 509us) server_negotiation.cc:407] Connection header received
0619 11:09:02.882487 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:02.882490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:02.882538 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:02.882628 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:02.883410 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:02.884161 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:02.885315 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:02.885637 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:02.885716 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:02.887299 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:02.887303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:02.887306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:02.887327 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:02.888917 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:02.889454 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:02.889458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:02.889460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:02.889535 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:02.889853 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:02.889856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:02.889857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:02.890034 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:02.890155 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:02.890440 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 11:09:02.890523 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:09:02.896136 4499 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:02.897294 4499 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:02.911693 4499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fa47ecbd6014603bb76d7d2cb337e7f: Generated new TSK 0
I20260619 11:09:02.912423 4499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:02.919781 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 4421
W20260619 11:09:02.925057 4406 connection.cc:535] client connection to 127.0.0.1:40235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40235): Connection reset by peer (error 104)
2026-06-19T11:09:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:02 dist-test-slave-8vcz krb5kdc[4520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:02 dist-test-slave-8vcz krb5kdc[4520](info): set up 2 sockets
Jun 19 11:09:02 dist-test-slave-8vcz krb5kdc[4520](info): commencing operation
krb5kdc: starting...
W20260619 11:09:03.943646 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:04 dist-test-slave-8vcz krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:04.058175 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:04.165176 4533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:04.165511 4533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:04.165619 4533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:04.169642 4533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:04.169691 4533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:04.169710 4533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:04.169726 4533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:04.169740 4533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:04.174916 4533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:04.175836 4533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:04.176735 4533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:04.184506 4538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:04.184660 4533 server_base.cc:1034] running on GCE node
W20260619 11:09:04.184523 4539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:04.184523 4541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:04.185350 4533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:04.186367 4533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:04.187543 4533 hybrid_clock.cc:648] HybridClock initialized: now 1781867344187532 us; error 23 us; skew 500 ppm
Jun 19 11:09:04 dist-test-slave-8vcz krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:04.190251 4533 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:04.191040 4533 webserver.cc:460] Webserver started at http://127.0.0.1:45417/ using document root <none> and password file <none>
I20260619 11:09:04.191556 4533 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:04.191642 4533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:04.191895 4533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:04.193825 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "0ecec17599584bd3bc0c08ab120f7aa1"
format_stamp: "Formatted at 2026-06-19 11:09:04 on dist-test-slave-8vcz"
server_key: "7c7ad73993817f6c5b48e2a69095acb1"
server_key_iv: "d6117a8a7782719716cead5487cc9478"
server_key_version: "encryptionkey@0"
I20260619 11:09:04.194370 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "0ecec17599584bd3bc0c08ab120f7aa1"
format_stamp: "Formatted at 2026-06-19 11:09:04 on dist-test-slave-8vcz"
server_key: "7c7ad73993817f6c5b48e2a69095acb1"
server_key_iv: "d6117a8a7782719716cead5487cc9478"
server_key_version: "encryptionkey@0"
I20260619 11:09:04.197856 4533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:09:04.200109 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:04.201346 4533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:04.201494 4533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "0ecec17599584bd3bc0c08ab120f7aa1"
format_stamp: "Formatted at 2026-06-19 11:09:04 on dist-test-slave-8vcz"
server_key: "7c7ad73993817f6c5b48e2a69095acb1"
server_key_iv: "d6117a8a7782719716cead5487cc9478"
server_key_version: "encryptionkey@0"
I20260619 11:09:04.201604 4533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:04.232597 4533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:04.235761 4533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:04.235958 4533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:04.242884 4533 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32991
I20260619 11:09:04.242934 4600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32991 every 8 connection(s)
I20260619 11:09:04.243852 4533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:04.246434 4601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:04.251734 4601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1: Bootstrap starting.
I20260619 11:09:04.253665 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 4533
I20260619 11:09:04.253808 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:04.254106 16628 external_mini_cluster.cc:1428] Setting key 5650fd13b9ab55467162c88cbabf869b
I20260619 11:09:04.254199 4601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:04.255018 4601 log.cc:824] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:04.256903 4601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1: No bootstrap required, opened a new log
I20260619 11:09:04.260442 4601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecec17599584bd3bc0c08ab120f7aa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } }
I20260619 11:09:04.260730 4601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:04.260792 4601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ecec17599584bd3bc0c08ab120f7aa1, State: Initialized, Role: FOLLOWER
I20260619 11:09:04.261191 4601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecec17599584bd3bc0c08ab120f7aa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } }
I20260619 11:09:04.261335 4601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:04.261406 4601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:04.261509 4601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:04.262357 4601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecec17599584bd3bc0c08ab120f7aa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } }
I20260619 11:09:04.262497 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:04.255192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41763 (local address 127.0.0.1:32991)
0619 11:09:04.255611 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:04.255624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:04.255655 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:09:04.256332 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:04.256348 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:04.256651 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:04.256965 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:04.257359 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:04.258279 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:04.259162 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:04.259632 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:04.259820 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:04.260669 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:04.260674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:04.260685 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:04.261207 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:04.261262 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:04.261284 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:04.261425 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:04.261659 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 11:09:04.261795 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"spinlock_wait_cycles":25472,"thread_start_us":157,"threads_started":1}
I20260619 11:09:04.262719 4601 leader_election.cc:302] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ecec17599584bd3bc0c08ab120f7aa1; no voters:
I20260619 11:09:04.263038 4601 leader_election.cc:288] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:04.263134 4606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:04.263319 4606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 0ecec17599584bd3bc0c08ab120f7aa1, State: Running, Role: LEADER
I20260619 11:09:04.264179 4601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:04.264349 4606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecec17599584bd3bc0c08ab120f7aa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } }
I20260619 11:09:04.265604 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:04.265908 4607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ecec17599584bd3bc0c08ab120f7aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecec17599584bd3bc0c08ab120f7aa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } } }
I20260619 11:09:04.266022 4607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:04.266309 4608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ecec17599584bd3bc0c08ab120f7aa1. Latest consensus state: current_term: 1 leader_uuid: "0ecec17599584bd3bc0c08ab120f7aa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecec17599584bd3bc0c08ab120f7aa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32991 } } }
I20260619 11:09:04.266592 4620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:04.266623 4608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:04.269277 4620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:04.274506 4620 catalog_manager.cc:1347] Generated new cluster ID: 80fed2182cac4285a4f32cd8ccc3da8a
I20260619 11:09:04.274567 4620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:04.276230 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:04.267894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35379 (local address 127.0.0.1:32991)
0619 11:09:04.268001 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:04.268005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:04.271959 (+ 3954us) server_negotiation.cc:407] Connection header received
0619 11:09:04.272127 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:04.272130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:04.272179 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:04.272267 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:04.272534 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:04.273206 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:04.274405 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:04.274726 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:04.274842 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:04.275764 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:04.275768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:04.275770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:04.275782 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:04.275814 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:04.275825 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:04.275921 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:04.276041 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 11:09:04.276072 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:09:04.300983 4620 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:04.301910 4620 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:04.310899 4620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ecec17599584bd3bc0c08ab120f7aa1: Generated new TSK 0
I20260619 11:09:04.311549 4620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:04.320319 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 4533
W20260619 11:09:04.325693 4518 connection.cc:535] client connection to 127.0.0.1:32991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32991): Connection reset by peer (error 104)
2026-06-19T11:09:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:04 dist-test-slave-8vcz krb5kdc[4632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:04 dist-test-slave-8vcz krb5kdc[4632](info): set up 2 sockets
Jun 19 11:09:04 dist-test-slave-8vcz krb5kdc[4632](info): commencing operation
krb5kdc: starting...
W20260619 11:09:05.351714 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:05 dist-test-slave-8vcz krb5kdc[4632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:05.465855 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:05.569444 4645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:05.569702 4645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:05.569777 4645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:05.573755 4645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:05.573833 4645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:05.573871 4645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:05.573926 4645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:05.573966 4645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:05.578893 4645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:05.579865 4645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:05.580776 4645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:05.588479 4653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:05.588507 4651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:05.588446 4650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:05.588779 4645 server_base.cc:1034] running on GCE node
I20260619 11:09:05.589311 4645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:05.590317 4645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:05.591507 4645 hybrid_clock.cc:648] HybridClock initialized: now 1781867345591501 us; error 39 us; skew 500 ppm
Jun 19 11:09:05 dist-test-slave-8vcz krb5kdc[4632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:05.594141 4645 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:05.595227 4645 webserver.cc:460] Webserver started at http://127.0.0.1:41235/ using document root <none> and password file <none>
I20260619 11:09:05.595760 4645 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:05.595829 4645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:05.595983 4645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:05.597744 4645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "2b3987c6a2b14daca841ce7d18246622"
format_stamp: "Formatted at 2026-06-19 11:09:05 on dist-test-slave-8vcz"
server_key: "88f202be8971df1518986fdf02678f22"
server_key_iv: "fd8ba94ca52325d7c7314426946361d8"
server_key_version: "encryptionkey@0"
I20260619 11:09:05.598227 4645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "2b3987c6a2b14daca841ce7d18246622"
format_stamp: "Formatted at 2026-06-19 11:09:05 on dist-test-slave-8vcz"
server_key: "88f202be8971df1518986fdf02678f22"
server_key_iv: "fd8ba94ca52325d7c7314426946361d8"
server_key_version: "encryptionkey@0"
I20260619 11:09:05.601574 4645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:05.603645 4660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:05.604772 4645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:05.604913 4645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "2b3987c6a2b14daca841ce7d18246622"
format_stamp: "Formatted at 2026-06-19 11:09:05 on dist-test-slave-8vcz"
server_key: "88f202be8971df1518986fdf02678f22"
server_key_iv: "fd8ba94ca52325d7c7314426946361d8"
server_key_version: "encryptionkey@0"
I20260619 11:09:05.605016 4645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:05.609210 4645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:05.611738 4645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:05.611930 4645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:05.618341 4645 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33941
I20260619 11:09:05.618394 4712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33941 every 8 connection(s)
I20260619 11:09:05.619194 4645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:05.620852 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 4645
I20260619 11:09:05.621014 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:05.621279 16628 external_mini_cluster.cc:1428] Setting key a2d82894a35bf53f32b245f5284da508
I20260619 11:09:05.621924 4713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:05.626502 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:05.622521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58317 (local address 127.0.0.1:33941)
0619 11:09:05.622905 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:05.622922 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:05.622960 (+ 38us) server_negotiation.cc:407] Connection header received
0619 11:09:05.623816 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:05.623837 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:05.624177 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:05.624519 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:05.625206 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:05.625212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:05.625227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:05.625238 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:05.625297 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:05.625323 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:05.625471 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:05.625543 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 11:09:05.625673 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260619 11:09:05.628569 4713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622: Bootstrap starting.
I20260619 11:09:05.629148 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:05.630650 4713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:05.631502 4713 log.cc:824] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:05.632658 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:05.631035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57769 (local address 127.0.0.1:33941)
0619 11:09:05.631162 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:05.631166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:05.631359 (+ 193us) server_negotiation.cc:407] Connection header received
0619 11:09:05.631468 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:05.631470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:05.631496 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:05.631558 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:05.632268 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:05.632271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:05.632274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:05.632284 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:05.632308 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:05.632319 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:05.632413 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:05.632515 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 11:09:05.632550 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 11:09:05.633603 4713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622: No bootstrap required, opened a new log
I20260619 11:09:05.637159 4713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3987c6a2b14daca841ce7d18246622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260619 11:09:05.637355 4713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:05.637432 4713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3987c6a2b14daca841ce7d18246622, State: Initialized, Role: FOLLOWER
I20260619 11:09:05.637809 4713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3987c6a2b14daca841ce7d18246622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260619 11:09:05.637954 4713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:05.638022 4713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:05.638120 4713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:05.638906 4713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3987c6a2b14daca841ce7d18246622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260619 11:09:05.639243 4713 leader_election.cc:302] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b3987c6a2b14daca841ce7d18246622; no voters:
I20260619 11:09:05.639519 4713 leader_election.cc:288] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:05.639679 4723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:05.639959 4723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [term 1 LEADER]: Becoming Leader. State: Replica: 2b3987c6a2b14daca841ce7d18246622, State: Running, Role: LEADER
I20260619 11:09:05.640321 4723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3987c6a2b14daca841ce7d18246622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260619 11:09:05.640674 4713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:05.641793 4724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b3987c6a2b14daca841ce7d18246622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3987c6a2b14daca841ce7d18246622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } } }
I20260619 11:09:05.641916 4724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:05.642272 4725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b3987c6a2b14daca841ce7d18246622. Latest consensus state: current_term: 1 leader_uuid: "2b3987c6a2b14daca841ce7d18246622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b3987c6a2b14daca841ce7d18246622" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } } }
I20260619 11:09:05.642344 4732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:05.642396 4725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:05.644981 4732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:05.649917 4732 catalog_manager.cc:1347] Generated new cluster ID: cb60cb8fa4434b33b522e27fada83cd1
I20260619 11:09:05.649986 4732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:05.673079 4732 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:05.673382 4732 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:05.683295 4732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b3987c6a2b14daca841ce7d18246622: Generated new TSK 0
I20260619 11:09:05.684020 4732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:05.693612 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 4645
2026-06-19T11:09:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:05 dist-test-slave-8vcz krb5kdc[4744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:05 dist-test-slave-8vcz krb5kdc[4744](info): set up 2 sockets
Jun 19 11:09:05 dist-test-slave-8vcz krb5kdc[4744](info): commencing operation
krb5kdc: starting...
W20260619 11:09:06.723896 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:06 dist-test-slave-8vcz krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:06.839932 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41107
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:06.942286 4757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:06.942545 4757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:06.942625 4757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:06.946471 4757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:06.946550 4757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:06.946594 4757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:06.946652 4757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:06.946691 4757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:06.951632 4757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:06.952558 4757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:06.953485 4757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:06.961345 4762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:06.961390 4765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:06.961370 4757 server_base.cc:1034] running on GCE node
W20260619 11:09:06.961369 4763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:06.962067 4757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:06.963068 4757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:06.964257 4757 hybrid_clock.cc:648] HybridClock initialized: now 1781867346964244 us; error 35 us; skew 500 ppm
Jun 19 11:09:06 dist-test-slave-8vcz krb5kdc[4744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:06.966799 4757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:06.967571 4757 webserver.cc:460] Webserver started at http://127.0.0.1:37549/ using document root <none> and password file <none>
I20260619 11:09:06.968104 4757 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:06.968199 4757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:06.968428 4757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:06.970271 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "4127b3904b2a4f0db355fdda4aa0603c"
format_stamp: "Formatted at 2026-06-19 11:09:06 on dist-test-slave-8vcz"
server_key: "5764f33426d2a240903e871b61059097"
server_key_iv: "cad79bfa831578056efcaf3d4f619a32"
server_key_version: "encryptionkey@0"
I20260619 11:09:06.970824 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "4127b3904b2a4f0db355fdda4aa0603c"
format_stamp: "Formatted at 2026-06-19 11:09:06 on dist-test-slave-8vcz"
server_key: "5764f33426d2a240903e871b61059097"
server_key_iv: "cad79bfa831578056efcaf3d4f619a32"
server_key_version: "encryptionkey@0"
I20260619 11:09:06.974298 4757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:06.976524 4772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:06.977581 4757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:06.977717 4757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "4127b3904b2a4f0db355fdda4aa0603c"
format_stamp: "Formatted at 2026-06-19 11:09:06 on dist-test-slave-8vcz"
server_key: "5764f33426d2a240903e871b61059097"
server_key_iv: "cad79bfa831578056efcaf3d4f619a32"
server_key_version: "encryptionkey@0"
I20260619 11:09:06.977823 4757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:06.994369 4757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:06.996896 4757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:06.997087 4757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:07.003835 4757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41107
I20260619 11:09:07.003846 4824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41107 every 8 connection(s)
I20260619 11:09:07.004707 4757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:07.007210 4825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:07.012169 4825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c: Bootstrap starting.
I20260619 11:09:07.014375 4825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:07.014786 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 4757
I20260619 11:09:07.014928 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:07.015177 16628 external_mini_cluster.cc:1428] Setting key 7d4ed91e0cf8886aba14ad314b2fbabd
I20260619 11:09:07.015156 4825 log.cc:824] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:07.017079 4825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c: No bootstrap required, opened a new log
I20260619 11:09:07.021060 4825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b3904b2a4f0db355fdda4aa0603c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41107 } }
I20260619 11:09:07.021265 4825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:07.021298 4825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4127b3904b2a4f0db355fdda4aa0603c, State: Initialized, Role: FOLLOWER
I20260619 11:09:07.021727 4825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b3904b2a4f0db355fdda4aa0603c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41107 } }
I20260619 11:09:07.021870 4825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:07.021958 4825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:07.022087 4825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:07.022868 4825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b3904b2a4f0db355fdda4aa0603c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41107 } }
I20260619 11:09:07.023234 4825 leader_election.cc:302] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4127b3904b2a4f0db355fdda4aa0603c; no voters:
I20260619 11:09:07.023554 4825 leader_election.cc:288] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:07.023645 4830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 11:09:07 dist-test-slave-8vcz krb5kdc[4744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:09:07.023880 4830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [term 1 LEADER]: Becoming Leader. State: Replica: 4127b3904b2a4f0db355fdda4aa0603c, State: Running, Role: LEADER
I20260619 11:09:07.024168 4830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b3904b2a4f0db355fdda4aa0603c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41107 } }
I20260619 11:09:07.024816 4825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:07.025316 4832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4127b3904b2a4f0db355fdda4aa0603c. Latest consensus state: current_term: 1 leader_uuid: "4127b3904b2a4f0db355fdda4aa0603c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b3904b2a4f0db355fdda4aa0603c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41107 } } }
I20260619 11:09:07.025430 4832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:07.025875 4835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:07.029316 4835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:07.029882 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:07.016164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42021 (local address 127.0.0.1:41107)
0619 11:09:07.016528 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:07.016544 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:07.016587 (+ 43us) server_negotiation.cc:407] Connection header received
0619 11:09:07.017248 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:07.017266 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:07.017565 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:07.017829 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:07.018679 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:07.019598 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:07.020853 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:07.021343 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:07.021503 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:07.024443 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:07.024449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:07.024464 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:07.024493 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:07.026605 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:07.027091 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:07.027097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:07.027102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:07.027225 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:07.027523 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:07.027526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:07.027528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:07.028047 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:07.028238 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:07.028514 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 11:09:07.028802 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":116,"threads_started":1}
I20260619 11:09:07.031215 4831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4127b3904b2a4f0db355fdda4aa0603c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4127b3904b2a4f0db355fdda4aa0603c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41107 } } }
I20260619 11:09:07.031388 4831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:07.032883 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:07.035470 4835 catalog_manager.cc:1347] Generated new cluster ID: c39b1d450d8c4a66a3b41641ec5b9780
I20260619 11:09:07.035545 4835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:07.043622 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:07.034892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54823 (local address 127.0.0.1:41107)
0619 11:09:07.034994 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:07.034998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:07.035222 (+ 224us) server_negotiation.cc:407] Connection header received
0619 11:09:07.035748 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:07.035751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:07.035798 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:07.035878 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:07.036661 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:07.037419 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:07.038545 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:07.038888 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:07.038968 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:07.040519 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:07.040523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:07.040525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:07.040548 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:07.042084 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:07.042534 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:07.042537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:07.042539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:07.042612 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:07.042886 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:07.042889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:07.042890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:07.043056 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:07.043161 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:07.043387 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:09:07.043484 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:09:07.063135 4835 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:07.064395 4835 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:07.071504 4835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4127b3904b2a4f0db355fdda4aa0603c: Generated new TSK 0
I20260619 11:09:07.072153 4835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:07.079262 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 4757
2026-06-19T11:09:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:07 dist-test-slave-8vcz krb5kdc[4856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:07 dist-test-slave-8vcz krb5kdc[4856](info): set up 2 sockets
Jun 19 11:09:07 dist-test-slave-8vcz krb5kdc[4856](info): commencing operation
krb5kdc: starting...
W20260619 11:09:08.107898 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:08 dist-test-slave-8vcz krb5kdc[4856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:08.224932 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:08.329401 4869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:08.329622 4869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:08.329704 4869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:08.334321 4869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:08.334376 4869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:08.334396 4869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:08.334412 4869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:08.334494 4869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:08.339447 4869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:08.340368 4869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:08.341303 4869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:08.349118 4875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:08.349226 4869 server_base.cc:1034] running on GCE node
W20260619 11:09:08.349166 4874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:08.349131 4877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:08.349896 4869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:08.350857 4869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:08.352037 4869 hybrid_clock.cc:648] HybridClock initialized: now 1781867348351998 us; error 51 us; skew 500 ppm
Jun 19 11:09:08 dist-test-slave-8vcz krb5kdc[4856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:08.358865 4869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:08.359771 4869 webserver.cc:460] Webserver started at http://127.0.0.1:42139/ using document root <none> and password file <none>
I20260619 11:09:08.360291 4869 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:08.360378 4869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:08.360595 4869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:08.362496 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "1bb350f03d5e4eecbb9a202c159054a8"
format_stamp: "Formatted at 2026-06-19 11:09:08 on dist-test-slave-8vcz"
server_key: "75c2ccef2836c09f50ffd4f2a8c26c93"
server_key_iv: "d3ad185f93b223cedec1febe03e8780c"
server_key_version: "encryptionkey@0"
I20260619 11:09:08.363044 4869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "1bb350f03d5e4eecbb9a202c159054a8"
format_stamp: "Formatted at 2026-06-19 11:09:08 on dist-test-slave-8vcz"
server_key: "75c2ccef2836c09f50ffd4f2a8c26c93"
server_key_iv: "d3ad185f93b223cedec1febe03e8780c"
server_key_version: "encryptionkey@0"
I20260619 11:09:08.366495 4869 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:08.368777 4884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:08.369910 4869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 11:09:08.370047 4869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "1bb350f03d5e4eecbb9a202c159054a8"
format_stamp: "Formatted at 2026-06-19 11:09:08 on dist-test-slave-8vcz"
server_key: "75c2ccef2836c09f50ffd4f2a8c26c93"
server_key_iv: "d3ad185f93b223cedec1febe03e8780c"
server_key_version: "encryptionkey@0"
I20260619 11:09:08.370150 4869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:08.386452 4869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:08.389171 4869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:08.389364 4869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:08.396515 4869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43345
I20260619 11:09:08.396541 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43345 every 8 connection(s)
I20260619 11:09:08.397398 4869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:08.399870 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 4869
I20260619 11:09:08.400009 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:08.400163 4937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:08.400323 16628 external_mini_cluster.cc:1428] Setting key 5fe8e6c5021ceab57ad5fed882e846b9
I20260619 11:09:08.406181 4937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8: Bootstrap starting.
I20260619 11:09:08.408437 4937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:08.408645 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:08.401616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40897 (local address 127.0.0.1:43345)
0619 11:09:08.402136 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:08.402149 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:08.402181 (+ 32us) server_negotiation.cc:407] Connection header received
0619 11:09:08.402759 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:08.402773 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:08.403079 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:08.403406 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:08.403805 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:08.404724 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:08.405668 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:08.406128 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:08.406296 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:08.407108 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:08.407113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:08.407124 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:08.407466 (+ 342us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:08.407513 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:08.407533 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:08.407639 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:08.407776 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 11:09:08.407972 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":149,"threads_started":1}
I20260619 11:09:08.409195 4937 log.cc:824] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:08.411067 4937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8: No bootstrap required, opened a new log
I20260619 11:09:08.411232 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:08.415303 4937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb350f03d5e4eecbb9a202c159054a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } }
I20260619 11:09:08.415580 4937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:08.415663 4937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb350f03d5e4eecbb9a202c159054a8, State: Initialized, Role: FOLLOWER
I20260619 11:09:08.416137 4937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb350f03d5e4eecbb9a202c159054a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } }
I20260619 11:09:08.416329 4937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:08.416399 4937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:08.416493 4937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:08.417073 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:08.413129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53519 (local address 127.0.0.1:43345)
0619 11:09:08.413229 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:08.413231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:08.413411 (+ 180us) server_negotiation.cc:407] Connection header received
0619 11:09:08.413510 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:08.413512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:08.413543 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:08.413604 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:08.413924 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:08.414385 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:08.415457 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:08.415814 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:08.415918 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:08.416687 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:08.416690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:08.416691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:08.416701 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:08.416722 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:08.416731 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:08.416808 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:08.416893 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 11:09:08.416936 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:09:08.417527 4937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb350f03d5e4eecbb9a202c159054a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } }
I20260619 11:09:08.417857 4937 leader_election.cc:302] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb350f03d5e4eecbb9a202c159054a8; no voters:
I20260619 11:09:08.418164 4937 leader_election.cc:288] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:08.418252 4947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:08.418498 4947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [term 1 LEADER]: Becoming Leader. State: Replica: 1bb350f03d5e4eecbb9a202c159054a8, State: Running, Role: LEADER
I20260619 11:09:08.418799 4947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb350f03d5e4eecbb9a202c159054a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } }
I20260619 11:09:08.419250 4937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:08.420282 4949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bb350f03d5e4eecbb9a202c159054a8. Latest consensus state: current_term: 1 leader_uuid: "1bb350f03d5e4eecbb9a202c159054a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb350f03d5e4eecbb9a202c159054a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } } }
I20260619 11:09:08.420635 4949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:08.420543 4948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bb350f03d5e4eecbb9a202c159054a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb350f03d5e4eecbb9a202c159054a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43345 } } }
I20260619 11:09:08.420975 4948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8 [sys.catalog]: This master's current role is: LEADER
W20260619 11:09:08.423797 4963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:08.423873 4963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:08.423944 4956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:08.424580 4956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:08.429945 4956 catalog_manager.cc:1347] Generated new cluster ID: 2696a8ae98844c00b1064818b65a821a
I20260619 11:09:08.430003 4956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:08.449925 4956 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:08.451043 4956 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:08.455183 4956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bb350f03d5e4eecbb9a202c159054a8: Generated new TSK 0
I20260619 11:09:08.455811 4956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:08.458963 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 4869
2026-06-19T11:09:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:08 dist-test-slave-8vcz krb5kdc[4968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:08 dist-test-slave-8vcz krb5kdc[4968](info): set up 2 sockets
Jun 19 11:09:08 dist-test-slave-8vcz krb5kdc[4968](info): commencing operation
krb5kdc: starting...
W20260619 11:09:09.491878 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:09 dist-test-slave-8vcz krb5kdc[4968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:09.622525 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:09.725622 4981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:09.725890 4981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:09.725976 4981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:09.730079 4981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:09.730158 4981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:09.730195 4981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:09.730250 4981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:09.730289 4981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:09.735370 4981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:09.736351 4981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:09.737296 4981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:09.745056 4986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:09.745144 4981 server_base.cc:1034] running on GCE node
W20260619 11:09:09.745056 4989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:09.745074 4987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:09.745796 4981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:09.746802 4981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:09.747988 4981 hybrid_clock.cc:648] HybridClock initialized: now 1781867349747970 us; error 28 us; skew 500 ppm
Jun 19 11:09:09 dist-test-slave-8vcz krb5kdc[4968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:09.750533 4981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:09.751349 4981 webserver.cc:460] Webserver started at http://127.0.0.1:39375/ using document root <none> and password file <none>
I20260619 11:09:09.751930 4981 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:09.752022 4981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:09.752249 4981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:09.754134 4981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "44742f42fb7548619824da7af82ca370"
format_stamp: "Formatted at 2026-06-19 11:09:09 on dist-test-slave-8vcz"
server_key: "cf78aeda7b637636dfa99a3810d12018"
server_key_iv: "63ae8f58f73c3e87c77f5451d8252d5c"
server_key_version: "encryptionkey@0"
I20260619 11:09:09.754676 4981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "44742f42fb7548619824da7af82ca370"
format_stamp: "Formatted at 2026-06-19 11:09:09 on dist-test-slave-8vcz"
server_key: "cf78aeda7b637636dfa99a3810d12018"
server_key_iv: "63ae8f58f73c3e87c77f5451d8252d5c"
server_key_version: "encryptionkey@0"
I20260619 11:09:09.758163 4981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 11:09:09.760385 4996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:09.761471 4981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:09.761605 4981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "44742f42fb7548619824da7af82ca370"
format_stamp: "Formatted at 2026-06-19 11:09:09 on dist-test-slave-8vcz"
server_key: "cf78aeda7b637636dfa99a3810d12018"
server_key_iv: "63ae8f58f73c3e87c77f5451d8252d5c"
server_key_version: "encryptionkey@0"
I20260619 11:09:09.761708 4981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:09.765990 4981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:09.768659 4981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:09.768846 4981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:09.775298 4981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35715
I20260619 11:09:09.775362 5048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35715 every 8 connection(s)
I20260619 11:09:09.776209 4981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:09.777325 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 4981
I20260619 11:09:09.777496 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:09.777804 16628 external_mini_cluster.cc:1428] Setting key e55284f051495c1cf583b0123afb0a32
I20260619 11:09:09.779060 5049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:09.782914 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:09.778995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55433 (local address 127.0.0.1:35715)
0619 11:09:09.779419 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:09.779436 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:09.779475 (+ 39us) server_negotiation.cc:407] Connection header received
0619 11:09:09.780267 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:09.780287 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:09.780606 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:09.780943 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:09.781616 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:09.781622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:09.781636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:09.781649 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:09.781706 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:09.781731 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:09.781873 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:09.781939 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 11:09:09.782077 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":160,"threads_started":1}
I20260619 11:09:09.785459 5049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370: Bootstrap starting.
I20260619 11:09:09.785552 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:09.787669 5049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:09.788460 5049 log.cc:824] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:09.789098 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:09.787390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54459 (local address 127.0.0.1:35715)
0619 11:09:09.787507 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:09.787513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:09.787703 (+ 190us) server_negotiation.cc:407] Connection header received
0619 11:09:09.787872 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:09.787877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:09.787915 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:09.787996 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:09.788709 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:09.788716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:09.788718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:09.788729 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:09.788758 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:09.788775 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:09.788870 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:09.788961 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 11:09:09.788987 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:09:09.790369 5049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370: No bootstrap required, opened a new log
I20260619 11:09:09.793704 5049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44742f42fb7548619824da7af82ca370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } }
I20260619 11:09:09.793939 5049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:09.794015 5049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44742f42fb7548619824da7af82ca370, State: Initialized, Role: FOLLOWER
I20260619 11:09:09.794389 5049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44742f42fb7548619824da7af82ca370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } }
I20260619 11:09:09.794530 5049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:09.794615 5049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:09.794710 5049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:09.795598 5049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44742f42fb7548619824da7af82ca370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } }
I20260619 11:09:09.795953 5049 leader_election.cc:302] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44742f42fb7548619824da7af82ca370; no voters:
I20260619 11:09:09.796263 5049 leader_election.cc:288] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:09.796379 5059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:09.796592 5059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [term 1 LEADER]: Becoming Leader. State: Replica: 44742f42fb7548619824da7af82ca370, State: Running, Role: LEADER
I20260619 11:09:09.796906 5059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44742f42fb7548619824da7af82ca370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } }
I20260619 11:09:09.797439 5049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:09.798451 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44742f42fb7548619824da7af82ca370. Latest consensus state: current_term: 1 leader_uuid: "44742f42fb7548619824da7af82ca370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44742f42fb7548619824da7af82ca370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } } }
I20260619 11:09:09.798588 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:09.798916 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44742f42fb7548619824da7af82ca370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44742f42fb7548619824da7af82ca370" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35715 } } }
I20260619 11:09:09.798979 5068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:09.799513 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:09.802507 5068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:09.807610 5068 catalog_manager.cc:1347] Generated new cluster ID: 075d4d3c06ba427e9b0c0783952fd79e
I20260619 11:09:09.807699 5068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:09.827914 5068 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:09.828254 5068 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:09.836979 5068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44742f42fb7548619824da7af82ca370: Generated new TSK 0
I20260619 11:09:09.837667 5068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:09.849884 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 4981
2026-06-19T11:09:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:09 dist-test-slave-8vcz krb5kdc[5080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:09 dist-test-slave-8vcz krb5kdc[5080](info): set up 2 sockets
Jun 19 11:09:09 dist-test-slave-8vcz krb5kdc[5080](info): commencing operation
krb5kdc: starting...
W20260619 11:09:10.891917 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:10 dist-test-slave-8vcz krb5kdc[5080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:11.006258 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:11.111631 5093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:11.111893 5093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:11.111971 5093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:11.115947 5093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:11.115993 5093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:11.116012 5093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:11.116067 5093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:11.116101 5093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:11.121132 5093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:11.122048 5093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:11.122987 5093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:11.131196 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:11.131160 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:11.131116 5101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:11.131475 5093 server_base.cc:1034] running on GCE node
I20260619 11:09:11.132079 5093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:11.133097 5093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:11.134285 5093 hybrid_clock.cc:648] HybridClock initialized: now 1781867351134267 us; error 29 us; skew 500 ppm
Jun 19 11:09:11 dist-test-slave-8vcz krb5kdc[5080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:11.137115 5093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:11.137965 5093 webserver.cc:460] Webserver started at http://127.0.0.1:43955/ using document root <none> and password file <none>
I20260619 11:09:11.138526 5093 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:11.138617 5093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:11.138840 5093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:11.140913 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "bbcab64446fb4e3095e8f5288c3dfc81"
format_stamp: "Formatted at 2026-06-19 11:09:11 on dist-test-slave-8vcz"
server_key: "1367f90035156d4a3dae47284310e476"
server_key_iv: "b59732d9b979e667d8b3fd8b5a5a6af0"
server_key_version: "encryptionkey@0"
I20260619 11:09:11.141470 5093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "bbcab64446fb4e3095e8f5288c3dfc81"
format_stamp: "Formatted at 2026-06-19 11:09:11 on dist-test-slave-8vcz"
server_key: "1367f90035156d4a3dae47284310e476"
server_key_iv: "b59732d9b979e667d8b3fd8b5a5a6af0"
server_key_version: "encryptionkey@0"
I20260619 11:09:11.145056 5093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:11.147374 5108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:11.148651 5093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:09:11.148789 5093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "bbcab64446fb4e3095e8f5288c3dfc81"
format_stamp: "Formatted at 2026-06-19 11:09:11 on dist-test-slave-8vcz"
server_key: "1367f90035156d4a3dae47284310e476"
server_key_iv: "b59732d9b979e667d8b3fd8b5a5a6af0"
server_key_version: "encryptionkey@0"
I20260619 11:09:11.148905 5093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:11.174736 5093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:11.177320 5093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:11.177511 5093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:11.184089 5093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41023
I20260619 11:09:11.184119 5160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41023 every 8 connection(s)
I20260619 11:09:11.184958 5093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:11.187456 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:11.191852 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5093
I20260619 11:09:11.192001 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:11.192305 16628 external_mini_cluster.cc:1428] Setting key 394dd32a1f3f476017846d02693ace5c
I20260619 11:09:11.193868 5161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81: Bootstrap starting.
I20260619 11:09:11.195968 5161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:11.196849 5161 log.cc:824] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:11.198726 5161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81: No bootstrap required, opened a new log
Jun 19 11:09:11 dist-test-slave-8vcz krb5kdc[5080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:09:11.202648 5161 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcab64446fb4e3095e8f5288c3dfc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260619 11:09:11.202883 5161 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:11.202941 5161 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcab64446fb4e3095e8f5288c3dfc81, State: Initialized, Role: FOLLOWER
I20260619 11:09:11.203420 5161 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcab64446fb4e3095e8f5288c3dfc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260619 11:09:11.203567 5161 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:11.203634 5161 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:11.203761 5161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:11.204829 5161 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcab64446fb4e3095e8f5288c3dfc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260619 11:09:11.205183 5161 leader_election.cc:302] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbcab64446fb4e3095e8f5288c3dfc81; no voters:
I20260619 11:09:11.205507 5161 leader_election.cc:288] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:11.205602 5166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:11.205852 5166 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [term 1 LEADER]: Becoming Leader. State: Replica: bbcab64446fb4e3095e8f5288c3dfc81, State: Running, Role: LEADER
I20260619 11:09:11.206207 5166 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcab64446fb4e3095e8f5288c3dfc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } }
I20260619 11:09:11.206786 5161 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:11.207070 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:11.193631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:49649 (local address 127.0.0.1:41023)
0619 11:09:11.194061 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:11.194076 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:11.194109 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:09:11.194701 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:11.194715 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:11.195029 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:11.195356 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:11.196190 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:11.197545 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:11.198395 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:11.199055 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:11.199214 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:11.201951 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:11.201956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:11.201967 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:11.201992 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:11.203991 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:11.204483 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:11.204491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:11.204497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:11.204637 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:11.204939 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:11.204943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:11.204944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:11.205420 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:11.205622 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:11.205977 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 11:09:11.206187 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260619 11:09:11.207288 5166 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbcab64446fb4e3095e8f5288c3dfc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcab64446fb4e3095e8f5288c3dfc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } } }
I20260619 11:09:11.207455 5166 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:11.207700 5166 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbcab64446fb4e3095e8f5288c3dfc81. Latest consensus state: current_term: 1 leader_uuid: "bbcab64446fb4e3095e8f5288c3dfc81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcab64446fb4e3095e8f5288c3dfc81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41023 } } }
I20260619 11:09:11.207839 5166 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:11.208125 5170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:11.210261 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:11.211264 5170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:11.216998 5170 catalog_manager.cc:1347] Generated new cluster ID: 4395bbec632f4a99b7aa26f7d1315665
I20260619 11:09:11.217056 5170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:11.224915 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:11.213557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:50707 (local address 127.0.0.1:41023)
0619 11:09:11.213659 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:11.213665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:11.213987 (+ 322us) server_negotiation.cc:407] Connection header received
0619 11:09:11.214138 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:11.214141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:11.214188 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:11.214268 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:11.214871 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:11.215368 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:11.216498 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:11.216850 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:11.216959 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:11.221365 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:11.221369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:11.221372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:11.221398 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:11.223004 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:11.223523 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:11.223526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:11.223528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:11.223608 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:11.223976 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:11.223979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:11.223981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:11.224169 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:11.224318 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:11.224666 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 11:09:11.224759 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 11:09:11.224985 5170 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:11.226153 5170 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:11.232977 5170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbcab64446fb4e3095e8f5288c3dfc81: Generated new TSK 0
I20260619 11:09:11.233601 5170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:11.237232 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5093
W20260619 11:09:11.241744 5078 connection.cc:535] client connection to 127.0.0.1:41023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41023): Connection reset by peer (error 104)
2026-06-19T11:09:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:11 dist-test-slave-8vcz krb5kdc[5192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:11 dist-test-slave-8vcz krb5kdc[5192](info): set up 2 sockets
Jun 19 11:09:11 dist-test-slave-8vcz krb5kdc[5192](info): commencing operation
krb5kdc: starting...
W20260619 11:09:12.331919 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:12 dist-test-slave-8vcz krb5kdc[5192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:12.449930 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:12.554836 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:12.555063 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:12.555128 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:12.559094 5205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:12.559146 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:12.559165 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:12.559181 5205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:12.559196 5205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:12.564225 5205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:12.565129 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:12.566004 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:12.574044 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:12.574050 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:12.574195 5205 server_base.cc:1034] running on GCE node
W20260619 11:09:12.574368 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:12.574911 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:12.575939 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:12.577091 5205 hybrid_clock.cc:648] HybridClock initialized: now 1781867352577067 us; error 37 us; skew 500 ppm
Jun 19 11:09:12 dist-test-slave-8vcz krb5kdc[5192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:12.579758 5205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:12.580587 5205 webserver.cc:460] Webserver started at http://127.0.0.1:39307/ using document root <none> and password file <none>
I20260619 11:09:12.581104 5205 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:12.581161 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:12.581362 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:12.583204 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "bdc3b16a1cf84117b51da15dfa8cc40c"
format_stamp: "Formatted at 2026-06-19 11:09:12 on dist-test-slave-8vcz"
server_key: "fb8d228e63bcfbb9789f8275ab3bb77b"
server_key_iv: "073fdce18f105f23e222c1845352f732"
server_key_version: "encryptionkey@0"
I20260619 11:09:12.583803 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "bdc3b16a1cf84117b51da15dfa8cc40c"
format_stamp: "Formatted at 2026-06-19 11:09:12 on dist-test-slave-8vcz"
server_key: "fb8d228e63bcfbb9789f8275ab3bb77b"
server_key_iv: "073fdce18f105f23e222c1845352f732"
server_key_version: "encryptionkey@0"
I20260619 11:09:12.587160 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 11:09:12.590070 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:12.591719 5205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:09:12.591857 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "bdc3b16a1cf84117b51da15dfa8cc40c"
format_stamp: "Formatted at 2026-06-19 11:09:12 on dist-test-slave-8vcz"
server_key: "fb8d228e63bcfbb9789f8275ab3bb77b"
server_key_iv: "073fdce18f105f23e222c1845352f732"
server_key_version: "encryptionkey@0"
I20260619 11:09:12.592033 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:12.605656 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:12.608340 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:12.608525 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:12.614905 5205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40211
I20260619 11:09:12.614939 5272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40211 every 8 connection(s)
I20260619 11:09:12.615859 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:12.618399 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:12.624068 5273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c: Bootstrap starting.
I20260619 11:09:12.625032 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5205
I20260619 11:09:12.625177 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:12.625468 16628 external_mini_cluster.cc:1428] Setting key d1a708a44996d19352b5a85f81119d51
I20260619 11:09:12.626775 5273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:12.627440 5273 log.cc:824] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:12.629185 5273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c: No bootstrap required, opened a new log
I20260619 11:09:12.632582 5273 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc3b16a1cf84117b51da15dfa8cc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40211 } }
I20260619 11:09:12.632789 5273 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:12.632912 5273 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc3b16a1cf84117b51da15dfa8cc40c, State: Initialized, Role: FOLLOWER
I20260619 11:09:12.633363 5273 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc3b16a1cf84117b51da15dfa8cc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40211 } }
I20260619 11:09:12.633509 5273 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:12.633591 5273 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:12.633687 5273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:12.633915 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:12.626486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:56241 (local address 127.0.0.1:40211)
0619 11:09:12.626926 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:12.626941 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:12.626972 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:09:12.627640 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:12.627657 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:12.628022 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:12.628332 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:12.628715 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:12.629693 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:12.630560 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:12.631021 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:12.631213 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:12.632150 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:12.632155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:12.632167 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:12.632520 (+ 353us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:12.632570 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:12.632591 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:12.632738 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:12.632912 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 11:09:12.633040 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":128,"threads_started":1}
I20260619 11:09:12.634555 5273 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc3b16a1cf84117b51da15dfa8cc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40211 } }
I20260619 11:09:12.634938 5273 leader_election.cc:302] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdc3b16a1cf84117b51da15dfa8cc40c; no voters:
I20260619 11:09:12.635258 5273 leader_election.cc:288] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:12.635354 5278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:12.635593 5278 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [term 1 LEADER]: Becoming Leader. State: Replica: bdc3b16a1cf84117b51da15dfa8cc40c, State: Running, Role: LEADER
I20260619 11:09:12.635948 5278 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc3b16a1cf84117b51da15dfa8cc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40211 } }
I20260619 11:09:12.636490 5273 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:12.636883 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:12.637771 5280 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdc3b16a1cf84117b51da15dfa8cc40c. Latest consensus state: current_term: 1 leader_uuid: "bdc3b16a1cf84117b51da15dfa8cc40c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc3b16a1cf84117b51da15dfa8cc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40211 } } }
I20260619 11:09:12.637779 5279 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdc3b16a1cf84117b51da15dfa8cc40c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdc3b16a1cf84117b51da15dfa8cc40c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40211 } } }
I20260619 11:09:12.637934 5280 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:12.637949 5279 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:12.638394 5291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:12.642256 5291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:12.644079 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:12.639163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:41145 (local address 127.0.0.1:40211)
0619 11:09:12.639281 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:12.639286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:12.639556 (+ 270us) server_negotiation.cc:407] Connection header received
0619 11:09:12.639774 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:12.639778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:12.639826 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:12.639875 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:12.640300 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:12.641086 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:12.642182 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:12.642526 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:12.642635 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:12.643521 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:12.643525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:12.643528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:12.643543 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:12.643571 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:12.643584 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:12.643726 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:12.643897 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 11:09:12.643933 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:09:12.647547 5291 catalog_manager.cc:1347] Generated new cluster ID: 8bccca70d7754972bd63d25c114773dd
I20260619 11:09:12.647606 5291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:12.663529 5291 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:12.664400 5291 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:12.673446 5291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdc3b16a1cf84117b51da15dfa8cc40c: Generated new TSK 0
I20260619 11:09:12.674070 5291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:12.678072 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5205
W20260619 11:09:12.682734 5190 connection.cc:535] client connection to 127.0.0.1:40211 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40211): Connection reset by peer (error 104)
2026-06-19T11:09:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:12 dist-test-slave-8vcz krb5kdc[5304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:12 dist-test-slave-8vcz krb5kdc[5304](info): set up 2 sockets
Jun 19 11:09:12 dist-test-slave-8vcz krb5kdc[5304](info): commencing operation
krb5kdc: starting...
W20260619 11:09:13.707893 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:13 dist-test-slave-8vcz krb5kdc[5304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:13.824738 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:13.928293 5317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:13.928541 5317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:13.928581 5317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:13.932601 5317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:13.932660 5317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:13.932679 5317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:13.932727 5317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:13.932758 5317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:13.937846 5317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:13.938889 5317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:13.939832 5317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:13.947860 5322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:13.947824 5323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:13.947824 5325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:13.948165 5317 server_base.cc:1034] running on GCE node
I20260619 11:09:13.948701 5317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:13.949740 5317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:13.950944 5317 hybrid_clock.cc:648] HybridClock initialized: now 1781867353950920 us; error 40 us; skew 500 ppm
Jun 19 11:09:13 dist-test-slave-8vcz krb5kdc[5304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:13.957267 5317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:13.958089 5317 webserver.cc:460] Webserver started at http://127.0.0.1:39295/ using document root <none> and password file <none>
I20260619 11:09:13.958632 5317 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:13.958724 5317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:13.958946 5317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:13.960947 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "b85d1368b83645a5804982f893944b4b"
format_stamp: "Formatted at 2026-06-19 11:09:13 on dist-test-slave-8vcz"
server_key: "ac773d4ab6a223b9e9999140a81a57a7"
server_key_iv: "dc8e8f04a404349e3607cba613a75493"
server_key_version: "encryptionkey@0"
I20260619 11:09:13.961514 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "b85d1368b83645a5804982f893944b4b"
format_stamp: "Formatted at 2026-06-19 11:09:13 on dist-test-slave-8vcz"
server_key: "ac773d4ab6a223b9e9999140a81a57a7"
server_key_iv: "dc8e8f04a404349e3607cba613a75493"
server_key_version: "encryptionkey@0"
I20260619 11:09:13.964973 5317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:13.967187 5332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:13.968397 5317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:13.968534 5317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "b85d1368b83645a5804982f893944b4b"
format_stamp: "Formatted at 2026-06-19 11:09:13 on dist-test-slave-8vcz"
server_key: "ac773d4ab6a223b9e9999140a81a57a7"
server_key_iv: "dc8e8f04a404349e3607cba613a75493"
server_key_version: "encryptionkey@0"
I20260619 11:09:13.968638 5317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:13.972971 5317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:13.976028 5317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:13.976212 5317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:13.982539 5317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39497
I20260619 11:09:13.982592 5384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39497 every 8 connection(s)
I20260619 11:09:13.983420 5317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:13.985977 5385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:13.989609 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5317
I20260619 11:09:13.989760 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:13.990033 16628 external_mini_cluster.cc:1428] Setting key 865d17609c880993c3b3bb6a82307d8d
I20260619 11:09:13.991304 5385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b: Bootstrap starting.
I20260619 11:09:13.993623 5385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:13.994472 5385 log.cc:824] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:13.994885 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:13.991240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:47521 (local address 127.0.0.1:39497)
0619 11:09:13.991643 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:13.991657 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:13.991730 (+ 73us) server_negotiation.cc:407] Connection header received
0619 11:09:13.992329 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:13.992344 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:13.992612 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:13.992873 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:13.993649 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:13.993653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:13.993664 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:13.993673 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:13.993742 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:13.993775 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:13.993872 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:13.994031 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 11:09:13.994198 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":144,"threads_started":1}
I20260619 11:09:13.996440 5385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b: No bootstrap required, opened a new log
I20260619 11:09:13.997604 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:14.000910 5385 raft_consensus.cc:348] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85d1368b83645a5804982f893944b4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } }
I20260619 11:09:14.001178 5385 raft_consensus.cc:374] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:14.001252 5385 raft_consensus.cc:729] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85d1368b83645a5804982f893944b4b, State: Initialized, Role: FOLLOWER
I20260619 11:09:14.001288 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:13.999753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:42067 (local address 127.0.0.1:39497)
0619 11:09:13.999872 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:13.999876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:14.000048 (+ 172us) server_negotiation.cc:407] Connection header received
0619 11:09:14.000127 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:14.000130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:14.000164 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:14.000238 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:14.000892 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:14.000896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:14.000899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:14.000911 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:14.000937 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:14.000950 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:14.001040 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:14.001098 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 11:09:14.001137 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 11:09:14.001699 5385 consensus_queue.cc:260] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85d1368b83645a5804982f893944b4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } }
I20260619 11:09:14.001821 5385 raft_consensus.cc:388] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:14.001910 5385 raft_consensus.cc:482] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:14.002007 5385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:14.002803 5385 raft_consensus.cc:504] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85d1368b83645a5804982f893944b4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } }
I20260619 11:09:14.003145 5385 leader_election.cc:302] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b85d1368b83645a5804982f893944b4b; no voters:
I20260619 11:09:14.003427 5385 leader_election.cc:288] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:14.003522 5395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:14.003737 5395 raft_consensus.cc:686] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [term 1 LEADER]: Becoming Leader. State: Replica: b85d1368b83645a5804982f893944b4b, State: Running, Role: LEADER
I20260619 11:09:14.004029 5395 consensus_queue.cc:237] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85d1368b83645a5804982f893944b4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } }
I20260619 11:09:14.004609 5385 sys_catalog.cc:564] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:14.005575 5396 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b85d1368b83645a5804982f893944b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85d1368b83645a5804982f893944b4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } } }
I20260619 11:09:14.005720 5396 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:14.005990 5397 sys_catalog.cc:455] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b85d1368b83645a5804982f893944b4b. Latest consensus state: current_term: 1 leader_uuid: "b85d1368b83645a5804982f893944b4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b85d1368b83645a5804982f893944b4b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } } }
I20260619 11:09:14.006106 5397 sys_catalog.cc:458] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:14.006145 5404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:14.009035 5404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:14.013823 5404 catalog_manager.cc:1347] Generated new cluster ID: 95b8b92bc61c460b941dd3ef8170c7fd
I20260619 11:09:14.013885 5404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:14.053889 5404 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:14.054191 5404 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:14.065631 5404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b85d1368b83645a5804982f893944b4b: Generated new TSK 0
I20260619 11:09:14.066256 5404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:14.077052 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5317
2026-06-19T11:09:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:14 dist-test-slave-8vcz krb5kdc[5416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:14 dist-test-slave-8vcz krb5kdc[5416](info): set up 2 sockets
Jun 19 11:09:14 dist-test-slave-8vcz krb5kdc[5416](info): commencing operation
krb5kdc: starting...
W20260619 11:09:15.107815 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:15 dist-test-slave-8vcz krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:15.226540 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:15.331933 5429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:15.332156 5429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:15.332229 5429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:15.336138 5429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:15.336186 5429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:15.336205 5429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:15.336221 5429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:15.336272 5429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:15.341246 5429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:15.342155 5429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:15.343079 5429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:15.350905 5435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:15.350873 5437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:15.351003 5434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:15.351150 5429 server_base.cc:1034] running on GCE node
I20260619 11:09:15.351784 5429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:15.352846 5429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:15.354009 5429 hybrid_clock.cc:648] HybridClock initialized: now 1781867355353992 us; error 29 us; skew 500 ppm
Jun 19 11:09:15 dist-test-slave-8vcz krb5kdc[5416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:15.356564 5429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:15.357357 5429 webserver.cc:460] Webserver started at http://127.0.0.1:43653/ using document root <none> and password file <none>
I20260619 11:09:15.357865 5429 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:15.357924 5429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:15.358155 5429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:15.360083 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "07a594fe882742908fd0de7055a7826a"
format_stamp: "Formatted at 2026-06-19 11:09:15 on dist-test-slave-8vcz"
server_key: "efac58cd12867b829390131a92ac7c99"
server_key_iv: "8130f78e40799f2d442afa97b572eeeb"
server_key_version: "encryptionkey@0"
I20260619 11:09:15.360620 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "07a594fe882742908fd0de7055a7826a"
format_stamp: "Formatted at 2026-06-19 11:09:15 on dist-test-slave-8vcz"
server_key: "efac58cd12867b829390131a92ac7c99"
server_key_iv: "8130f78e40799f2d442afa97b572eeeb"
server_key_version: "encryptionkey@0"
I20260619 11:09:15.364001 5429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:15.366170 5444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:15.367264 5429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:15.367389 5429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "07a594fe882742908fd0de7055a7826a"
format_stamp: "Formatted at 2026-06-19 11:09:15 on dist-test-slave-8vcz"
server_key: "efac58cd12867b829390131a92ac7c99"
server_key_iv: "8130f78e40799f2d442afa97b572eeeb"
server_key_version: "encryptionkey@0"
I20260619 11:09:15.367489 5429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:15.386462 5429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:15.389293 5429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:15.389493 5429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:15.396682 5429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45911
I20260619 11:09:15.396724 5496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45911 every 8 connection(s)
I20260619 11:09:15.397588 5429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:15.400128 5497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:15.401908 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5429
I20260619 11:09:15.402029 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:15.402318 16628 external_mini_cluster.cc:1428] Setting key c58672e738ac51a8b9ba3930b88656b3
I20260619 11:09:15.406095 5497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a: Bootstrap starting.
I20260619 11:09:15.408185 5497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:15.408859 5497 log.cc:824] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:09:15 dist-test-slave-8vcz krb5kdc[5416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 11:09:15.410889 5497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a: No bootstrap required, opened a new log
I20260619 11:09:15.415103 5497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a594fe882742908fd0de7055a7826a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260619 11:09:15.415333 5497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:15.415412 5497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a594fe882742908fd0de7055a7826a, State: Initialized, Role: FOLLOWER
I20260619 11:09:15.415897 5497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a594fe882742908fd0de7055a7826a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260619 11:09:15.416018 5497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:15.416100 5497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:15.416181 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:15.403620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:33443 (local address 127.0.0.1:45911)
0619 11:09:15.404044 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:15.404058 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:15.404091 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:09:15.404692 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:15.404707 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:15.405025 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:15.405336 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:15.406119 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:15.407107 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:15.407995 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:15.408488 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:15.408653 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:15.411465 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:15.411470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:15.411482 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:15.411506 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:15.413177 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:15.413745 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:15.413750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:15.413754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:15.413856 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:15.414214 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:15.414216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:15.414217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:15.414561 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:15.414724 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:15.415024 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 11:09:15.415293 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":140,"threads_started":1}
I20260619 11:09:15.416210 5497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:15.417070 5497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a594fe882742908fd0de7055a7826a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260619 11:09:15.417398 5497 leader_election.cc:302] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a594fe882742908fd0de7055a7826a; no voters:
I20260619 11:09:15.417701 5497 leader_election.cc:288] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:15.417765 5502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:15.417966 5502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [term 1 LEADER]: Becoming Leader. State: Replica: 07a594fe882742908fd0de7055a7826a, State: Running, Role: LEADER
I20260619 11:09:15.418282 5502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a594fe882742908fd0de7055a7826a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } }
I20260619 11:09:15.418879 5497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:15.419039 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:15.419370 5504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07a594fe882742908fd0de7055a7826a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a594fe882742908fd0de7055a7826a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260619 11:09:15.419546 5504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:15.419873 5503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a594fe882742908fd0de7055a7826a. Latest consensus state: current_term: 1 leader_uuid: "07a594fe882742908fd0de7055a7826a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a594fe882742908fd0de7055a7826a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45911 } } }
I20260619 11:09:15.420017 5508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:15.420032 5503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:15.423130 5508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:15.429117 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:15.421214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:41917 (local address 127.0.0.1:45911)
0619 11:09:15.421307 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:15.421313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:15.421644 (+ 331us) server_negotiation.cc:407] Connection header received
0619 11:09:15.421787 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:15.421790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:15.421836 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:15.421928 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:15.422634 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:15.423139 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:15.424162 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:15.424411 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:15.424515 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:15.425846 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:15.425850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:15.425852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:15.425875 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:15.427185 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:15.427645 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:15.427648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:15.427650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:15.427754 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:15.428161 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:15.428166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:15.428168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:15.428414 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:15.428564 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:15.428861 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 11:09:15.428956 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 11:09:15.431313 5508 catalog_manager.cc:1347] Generated new cluster ID: 13e0b2c2f63043fbbdabcc2b9197b82e
I20260619 11:09:15.431376 5508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:15.447382 5508 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:15.448570 5508 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:15.454684 5508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a594fe882742908fd0de7055a7826a: Generated new TSK 0
I20260619 11:09:15.455365 5508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:15.461057 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5429
W20260619 11:09:15.466259 5414 connection.cc:535] client connection to 127.0.0.1:45911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45911): Connection reset by peer (error 104)
2026-06-19T11:09:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:15 dist-test-slave-8vcz krb5kdc[5528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:15 dist-test-slave-8vcz krb5kdc[5528](info): set up 2 sockets
Jun 19 11:09:15 dist-test-slave-8vcz krb5kdc[5528](info): commencing operation
krb5kdc: starting...
W20260619 11:09:16.483880 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:16 dist-test-slave-8vcz krb5kdc[5528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:16.598714 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44901
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:16.703686 5541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:16.703962 5541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:16.704034 5541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:16.707985 5541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:16.708056 5541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:16.708091 5541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:16.708145 5541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:16.708184 5541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:16.713333 5541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:16.714306 5541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:16.715262 5541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:16.723169 5549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:16.723166 5546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:16.723207 5547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:16.723520 5541 server_base.cc:1034] running on GCE node
I20260619 11:09:16.724277 5541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:16.725271 5541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:16.726434 5541 hybrid_clock.cc:648] HybridClock initialized: now 1781867356726406 us; error 40 us; skew 500 ppm
Jun 19 11:09:16 dist-test-slave-8vcz krb5kdc[5528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:16.729040 5541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:16.729794 5541 webserver.cc:460] Webserver started at http://127.0.0.1:40417/ using document root <none> and password file <none>
I20260619 11:09:16.730283 5541 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:16.730341 5541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:16.730496 5541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:16.732487 5541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "570ef86768b74768b675abff82fee872"
format_stamp: "Formatted at 2026-06-19 11:09:16 on dist-test-slave-8vcz"
server_key: "3712d730ee1ea47ac3ac8d9270816171"
server_key_iv: "892927bc8ed5d5c0ed3865ea9d417fe5"
server_key_version: "encryptionkey@0"
I20260619 11:09:16.732976 5541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "570ef86768b74768b675abff82fee872"
format_stamp: "Formatted at 2026-06-19 11:09:16 on dist-test-slave-8vcz"
server_key: "3712d730ee1ea47ac3ac8d9270816171"
server_key_iv: "892927bc8ed5d5c0ed3865ea9d417fe5"
server_key_version: "encryptionkey@0"
I20260619 11:09:16.736529 5541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 11:09:16.738837 5556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:16.740123 5541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:16.740228 5541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "570ef86768b74768b675abff82fee872"
format_stamp: "Formatted at 2026-06-19 11:09:16 on dist-test-slave-8vcz"
server_key: "3712d730ee1ea47ac3ac8d9270816171"
server_key_iv: "892927bc8ed5d5c0ed3865ea9d417fe5"
server_key_version: "encryptionkey@0"
I20260619 11:09:16.740312 5541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:16.763386 5541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:16.766607 5541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:16.766753 5541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:16.773379 5541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44901
I20260619 11:09:16.773447 5608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44901 every 8 connection(s)
I20260619 11:09:16.774336 5541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:16.777064 5609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:16.782302 5609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872: Bootstrap starting.
I20260619 11:09:16.784192 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5541
I20260619 11:09:16.784368 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:16.784657 16628 external_mini_cluster.cc:1428] Setting key 1d38fd1ac4348e50e986a7b85aab4b5b
I20260619 11:09:16.784782 5609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:16.785681 5609 log.cc:824] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:16.787875 5609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872: No bootstrap required, opened a new log
I20260619 11:09:16.792179 5609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570ef86768b74768b675abff82fee872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } }
I20260619 11:09:16.792402 5609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:16.792438 5609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570ef86768b74768b675abff82fee872, State: Initialized, Role: FOLLOWER
I20260619 11:09:16.792915 5609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570ef86768b74768b675abff82fee872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } }
I20260619 11:09:16.793085 5609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:16.793166 5609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:16.793287 5609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:16.794234 5609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570ef86768b74768b675abff82fee872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } }
I20260619 11:09:16.794576 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:16.785819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:44389 (local address 127.0.0.1:44901)
0619 11:09:16.786319 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:16.786335 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:16.786370 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:09:16.787237 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:16.787259 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:16.787656 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:16.788058 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:16.788471 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:16.789945 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:16.790810 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:16.791548 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:16.791739 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:16.792705 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:16.792712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:16.792730 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:16.793207 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:16.793289 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:16.793319 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:16.793480 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:16.793647 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 11:09:16.793798 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"spinlock_wait_cycles":15744,"thread_start_us":200,"threads_started":1}
I20260619 11:09:16.794570 5609 leader_election.cc:302] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 570ef86768b74768b675abff82fee872; no voters:
I20260619 11:09:16.794951 5609 leader_election.cc:288] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:16.795089 5614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:16.795320 5614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [term 1 LEADER]: Becoming Leader. State: Replica: 570ef86768b74768b675abff82fee872, State: Running, Role: LEADER
I20260619 11:09:16.795652 5614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570ef86768b74768b675abff82fee872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } }
I20260619 11:09:16.796168 5609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:16.797451 5615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "570ef86768b74768b675abff82fee872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570ef86768b74768b675abff82fee872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } } }
I20260619 11:09:16.797551 5615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:16.797992 5624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:16.798312 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:16.798293 5616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 570ef86768b74768b675abff82fee872. Latest consensus state: current_term: 1 leader_uuid: "570ef86768b74768b675abff82fee872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "570ef86768b74768b675abff82fee872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44901 } } }
I20260619 11:09:16.798831 5616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:16.801506 5624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:16.805286 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:16.800427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:56829 (local address 127.0.0.1:44901)
0619 11:09:16.800550 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:16.800554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:16.800937 (+ 383us) server_negotiation.cc:407] Connection header received
0619 11:09:16.801029 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:16.801032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:16.801077 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:16.801159 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:16.801450 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:16.802174 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:16.803322 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:16.803639 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:16.803763 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:16.804772 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:16.804776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:16.804779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:16.804792 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:16.804829 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:16.804844 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:16.804940 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:16.805124 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 11:09:16.805158 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 11:09:16.809120 5624 catalog_manager.cc:1347] Generated new cluster ID: ea77697df66640a995c12a6158997db5
I20260619 11:09:16.809181 5624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:16.820427 5624 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:16.821601 5624 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:16.829960 5624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 570ef86768b74768b675abff82fee872: Generated new TSK 0
I20260619 11:09:16.830559 5624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:16.842062 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5541
W20260619 11:09:16.847657 5526 connection.cc:535] client connection to 127.0.0.1:44901 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44901): Connection reset by peer (error 104)
2026-06-19T11:09:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:16 dist-test-slave-8vcz krb5kdc[5640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:16 dist-test-slave-8vcz krb5kdc[5640](info): set up 2 sockets
Jun 19 11:09:16 dist-test-slave-8vcz krb5kdc[5640](info): commencing operation
krb5kdc: starting...
W20260619 11:09:17.879853 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:17 dist-test-slave-8vcz krb5kdc[5640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 11:09:17.992993 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:18.097592 5653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:18.097867 5653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:18.097960 5653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:18.101884 5653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:18.101965 5653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:18.102010 5653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:18.102066 5653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:18.102104 5653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:18.107019 5653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:18.108002 5653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:18.108916 5653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:18.116988 5661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:18.117010 5658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:18.117017 5659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:18.117198 5653 server_base.cc:1034] running on GCE node
I20260619 11:09:18.117718 5653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:18.118707 5653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:18.119859 5653 hybrid_clock.cc:648] HybridClock initialized: now 1781867358119844 us; error 27 us; skew 500 ppm
Jun 19 11:09:18 dist-test-slave-8vcz krb5kdc[5640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:18.122367 5653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 11:09:18.123201 5653 webserver.cc:460] Webserver started at http://127.0.0.1:45245/ using document root <none> and password file <none>
I20260619 11:09:18.123768 5653 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:18.123831 5653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:18.124066 5653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:18.125942 5653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "5685536687e84ed2bf954d3c89c7fbee"
format_stamp: "Formatted at 2026-06-19 11:09:18 on dist-test-slave-8vcz"
server_key: "4ddab2e4bea10c2dee667e89f1d5d14c"
server_key_iv: "24df9e291509f5abc34ae21882c52c00"
server_key_version: "encryptionkey@0"
I20260619 11:09:18.126466 5653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "5685536687e84ed2bf954d3c89c7fbee"
format_stamp: "Formatted at 2026-06-19 11:09:18 on dist-test-slave-8vcz"
server_key: "4ddab2e4bea10c2dee667e89f1d5d14c"
server_key_iv: "24df9e291509f5abc34ae21882c52c00"
server_key_version: "encryptionkey@0"
I20260619 11:09:18.129875 5653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:09:18.132107 5668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:18.133240 5653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:18.133373 5653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "5685536687e84ed2bf954d3c89c7fbee"
format_stamp: "Formatted at 2026-06-19 11:09:18 on dist-test-slave-8vcz"
server_key: "4ddab2e4bea10c2dee667e89f1d5d14c"
server_key_iv: "24df9e291509f5abc34ae21882c52c00"
server_key_version: "encryptionkey@0"
I20260619 11:09:18.133478 5653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:18.137741 5653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:18.140507 5653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:18.140693 5653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:18.147293 5653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45675
I20260619 11:09:18.147305 5720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45675 every 8 connection(s)
I20260619 11:09:18.148186 5653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:18.150770 5721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:18.156072 5721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee: Bootstrap starting.
I20260619 11:09:18.158399 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5653
I20260619 11:09:18.158447 5721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:18.158553 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:18.158859 16628 external_mini_cluster.cc:1428] Setting key 67f098ce948b2607c44c54a3dbfffb66
I20260619 11:09:18.159307 5721 log.cc:824] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:18.161243 5721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee: No bootstrap required, opened a new log
I20260619 11:09:18.163707 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:18.160133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:35675 (local address 127.0.0.1:45675)
0619 11:09:18.160535 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:18.160558 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:18.160591 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:09:18.161185 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:18.161201 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:18.161510 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:18.161770 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:18.162544 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:18.162549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:18.162559 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:18.162568 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:18.162618 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:18.162636 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:18.162718 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:18.162832 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 11:09:18.163011 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":125,"threads_started":1}
I20260619 11:09:18.165287 5721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685536687e84ed2bf954d3c89c7fbee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260619 11:09:18.165572 5721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:18.165637 5721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5685536687e84ed2bf954d3c89c7fbee, State: Initialized, Role: FOLLOWER
I20260619 11:09:18.166097 5721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685536687e84ed2bf954d3c89c7fbee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260619 11:09:18.166241 5721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:18.166309 5721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:18.166354 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:18.166410 5721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:18.167172 5721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685536687e84ed2bf954d3c89c7fbee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260619 11:09:18.167536 5721 leader_election.cc:302] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5685536687e84ed2bf954d3c89c7fbee; no voters:
I20260619 11:09:18.167867 5721 leader_election.cc:288] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:18.167928 5730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:18.168152 5730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [term 1 LEADER]: Becoming Leader. State: Replica: 5685536687e84ed2bf954d3c89c7fbee, State: Running, Role: LEADER
I20260619 11:09:18.168465 5730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685536687e84ed2bf954d3c89c7fbee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260619 11:09:18.169050 5721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:18.169618 5730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5685536687e84ed2bf954d3c89c7fbee. Latest consensus state: current_term: 1 leader_uuid: "5685536687e84ed2bf954d3c89c7fbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685536687e84ed2bf954d3c89c7fbee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } } }
I20260619 11:09:18.169767 5730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:18.170033 5731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5685536687e84ed2bf954d3c89c7fbee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5685536687e84ed2bf954d3c89c7fbee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } } }
I20260619 11:09:18.170197 5731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:18.170200 5738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:18.171028 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:18.168578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.110:35059 (local address 127.0.0.1:45675)
0619 11:09:18.168660 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:18.168663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:18.168886 (+ 223us) server_negotiation.cc:407] Connection header received
0619 11:09:18.169039 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:18.169041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:18.169070 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:18.169169 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:18.170591 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:18.170595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:18.170597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:18.170608 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:18.170634 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:18.170646 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:18.170702 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:18.170871 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 11:09:18.170908 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 11:09:18.173323 5738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:18.178241 5738 catalog_manager.cc:1347] Generated new cluster ID: 9afb3b196a524e39a5853a6d9ca85e02
I20260619 11:09:18.178350 5738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:18.189184 5738 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:18.189489 5738 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:18.198031 5738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5685536687e84ed2bf954d3c89c7fbee: Generated new TSK 0
I20260619 11:09:18.198623 5738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:18.203846 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5653
2026-06-19T11:09:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:18 dist-test-slave-8vcz krb5kdc[5752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:18 dist-test-slave-8vcz krb5kdc[5752](info): set up 2 sockets
Jun 19 11:09:18 dist-test-slave-8vcz krb5kdc[5752](info): commencing operation
krb5kdc: starting...
W20260619 11:09:19.235862 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:19 dist-test-slave-8vcz krb5kdc[5752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:09:19.352092 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:45499
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:37335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:19.456596 5765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:19.456856 5765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:19.456933 5765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:19.460809 5765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:19.460889 5765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:19.460933 5765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:19.460992 5765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:19.461031 5765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:19.465991 5765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:37335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:45499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:19.466958 5765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:19.467902 5765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:19.475723 5773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:19.475720 5771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:19.475904 5765 server_base.cc:1034] running on GCE node
W20260619 11:09:19.475739 5770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:19.476495 5765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:19.477540 5765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:19.478722 5765 hybrid_clock.cc:648] HybridClock initialized: now 1781867359478694 us; error 42 us; skew 500 ppm
Jun 19 11:09:19 dist-test-slave-8vcz krb5kdc[5752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867359, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:19.485921 5765 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:09:19.486763 5765 webserver.cc:460] Webserver started at http://127.16.61.62:41253/ using document root <none> and password file <none>
I20260619 11:09:19.487308 5765 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:19.487402 5765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:19.487637 5765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:19.489605 5765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "aecb8ea7dd1946348183fa1bbf633fdb"
format_stamp: "Formatted at 2026-06-19 11:09:19 on dist-test-slave-8vcz"
server_key: "81efdb67b8fe10fa02c4be442aa0260f"
server_key_iv: "3da159261c151466d537eb2218a3fa8a"
server_key_version: "encryptionkey@0"
I20260619 11:09:19.490163 5765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "aecb8ea7dd1946348183fa1bbf633fdb"
format_stamp: "Formatted at 2026-06-19 11:09:19 on dist-test-slave-8vcz"
server_key: "81efdb67b8fe10fa02c4be442aa0260f"
server_key_iv: "3da159261c151466d537eb2218a3fa8a"
server_key_version: "encryptionkey@0"
I20260619 11:09:19.493837 5765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 11:09:19.496171 5780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:19.497339 5765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:19.497488 5765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "aecb8ea7dd1946348183fa1bbf633fdb"
format_stamp: "Formatted at 2026-06-19 11:09:19 on dist-test-slave-8vcz"
server_key: "81efdb67b8fe10fa02c4be442aa0260f"
server_key_iv: "3da159261c151466d537eb2218a3fa8a"
server_key_version: "encryptionkey@0"
I20260619 11:09:19.497598 5765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:19.528631 5765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:19.531456 5765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:19.531657 5765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:19.538604 5765 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:45499
I20260619 11:09:19.538630 5832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:45499 every 8 connection(s)
I20260619 11:09:19.539512 5765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:19.542778 5833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:19.547853 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5765
I20260619 11:09:19.548019 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:19.548334 16628 external_mini_cluster.cc:1428] Setting key abc5f14d92d43ad028ee946e008a0c25
I20260619 11:09:19.552301 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb: Bootstrap starting.
I20260619 11:09:19.555588 5833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:19.556686 5833 log.cc:824] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:19.558995 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb: No bootstrap required, opened a new log
Jun 19 11:09:19 dist-test-slave-8vcz krb5kdc[5752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:19.563594 5833 raft_consensus.cc:348] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecb8ea7dd1946348183fa1bbf633fdb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45499 } }
I20260619 11:09:19.563921 5833 raft_consensus.cc:374] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:19.563980 5833 raft_consensus.cc:729] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecb8ea7dd1946348183fa1bbf633fdb, State: Initialized, Role: FOLLOWER
I20260619 11:09:19.564495 5833 consensus_queue.cc:260] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecb8ea7dd1946348183fa1bbf633fdb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45499 } }
I20260619 11:09:19.564705 5833 raft_consensus.cc:388] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:19.564780 5833 raft_consensus.cc:482] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:19.564891 5833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:19.566181 5833 raft_consensus.cc:504] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecb8ea7dd1946348183fa1bbf633fdb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45499 } }
I20260619 11:09:19.566574 5833 leader_election.cc:302] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aecb8ea7dd1946348183fa1bbf633fdb; no voters:
I20260619 11:09:19.566922 5833 leader_election.cc:288] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:19.567041 5838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:19.567291 5838 raft_consensus.cc:686] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [term 1 LEADER]: Becoming Leader. State: Replica: aecb8ea7dd1946348183fa1bbf633fdb, State: Running, Role: LEADER
I20260619 11:09:19.567646 5838 consensus_queue.cc:237] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecb8ea7dd1946348183fa1bbf633fdb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45499 } }
I20260619 11:09:19.568231 5833 sys_catalog.cc:564] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:19.570201 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:19.550374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38517 (local address 127.16.61.62:45499)
0619 11:09:19.550857 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:19.550880 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:19.550929 (+ 49us) server_negotiation.cc:407] Connection header received
0619 11:09:19.551966 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:19.551993 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:19.552511 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:19.552896 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:19.553822 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:19.555654 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:19.557076 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:19.557894 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:19.558110 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:19.561654 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:19.561660 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:19.561676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:19.561706 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:19.564620 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:19.565277 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:19.565286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:19.565294 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:19.565444 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:19.565868 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:19.565872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:19.565874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:19.566493 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:19.566708 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:19.568866 (+ 2158us) server_negotiation.cc:299] Negotiation successful
0619 11:09:19.569101 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
I20260619 11:09:19.571033 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader aecb8ea7dd1946348183fa1bbf633fdb. Latest consensus state: current_term: 1 leader_uuid: "aecb8ea7dd1946348183fa1bbf633fdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecb8ea7dd1946348183fa1bbf633fdb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45499 } } }
I20260619 11:09:19.571151 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:19.571445 5839 sys_catalog.cc:455] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aecb8ea7dd1946348183fa1bbf633fdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aecb8ea7dd1946348183fa1bbf633fdb" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45499 } } }
I20260619 11:09:19.571530 5839 sys_catalog.cc:458] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:19.572000 5853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:19.574182 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:19.575131 5853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:19.580802 5853 catalog_manager.cc:1347] Generated new cluster ID: a87e953cfae0405e8959556117a51883
I20260619 11:09:19.580857 5853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:19.587642 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:19.576576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44311 (local address 127.16.61.62:45499)
0619 11:09:19.576677 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:19.576681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:19.578883 (+ 2202us) server_negotiation.cc:407] Connection header received
0619 11:09:19.579070 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:19.579073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:19.579120 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:19.579180 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:19.580124 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:19.580865 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:19.582114 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:19.582477 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:19.582542 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:19.584211 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:19.584215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:19.584217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:19.584240 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:19.585824 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:19.586353 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:19.586356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:19.586358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:19.586431 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:19.586751 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:19.586754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:19.586756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:19.586931 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:19.587036 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:19.587402 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 11:09:19.587500 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:09:19.593166 5853 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:19.593988 5853 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:19.601512 5853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aecb8ea7dd1946348183fa1bbf633fdb: Generated new TSK 0
I20260619 11:09:19.602171 5853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:19.608471 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5765
W20260619 11:09:19.613824 5750 connection.cc:535] client connection to 127.16.61.62:45499 recv error: Network error: failed to read from TLS socket (remote: 127.16.61.62:45499): Connection reset by peer (error 104)
2026-06-19T11:09:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:19 dist-test-slave-8vcz krb5kdc[5864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:19 dist-test-slave-8vcz krb5kdc[5864](info): set up 2 sockets
Jun 19 11:09:19 dist-test-slave-8vcz krb5kdc[5864](info): commencing operation
krb5kdc: starting...
W20260619 11:09:20.644256 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:20 dist-test-slave-8vcz krb5kdc[5864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:09:20.762367 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:33439
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:39133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:20.868256 5877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:20.868506 5877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:20.868547 5877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:20.872457 5877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:20.872509 5877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:20.872529 5877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:20.872545 5877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:20.872560 5877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:20.877640 5877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:33439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:20.878583 5877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:20.879508 5877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:20.887650 5883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:20.887688 5877 server_base.cc:1034] running on GCE node
W20260619 11:09:20.887650 5885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:20.887694 5882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:20.888514 5877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:20.889565 5877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:20.890749 5877 hybrid_clock.cc:648] HybridClock initialized: now 1781867360890726 us; error 38 us; skew 500 ppm
Jun 19 11:09:20 dist-test-slave-8vcz krb5kdc[5864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867360, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:20.893429 5877 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:09:20.894248 5877 webserver.cc:460] Webserver started at http://127.16.61.62:44685/ using document root <none> and password file <none>
I20260619 11:09:20.894783 5877 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:20.894870 5877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:20.895095 5877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:20.897110 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "cd2e7323ee00473689da2a7295f7a64b"
format_stamp: "Formatted at 2026-06-19 11:09:20 on dist-test-slave-8vcz"
server_key: "e62efaa3476b53e5aacf6eea4449ed11"
server_key_iv: "7e923b17f1d08d56958ded27a26da507"
server_key_version: "encryptionkey@0"
I20260619 11:09:20.897660 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "cd2e7323ee00473689da2a7295f7a64b"
format_stamp: "Formatted at 2026-06-19 11:09:20 on dist-test-slave-8vcz"
server_key: "e62efaa3476b53e5aacf6eea4449ed11"
server_key_iv: "7e923b17f1d08d56958ded27a26da507"
server_key_version: "encryptionkey@0"
I20260619 11:09:20.901400 5877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:20.903931 5892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:20.905138 5877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:20.905277 5877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "cd2e7323ee00473689da2a7295f7a64b"
format_stamp: "Formatted at 2026-06-19 11:09:20 on dist-test-slave-8vcz"
server_key: "e62efaa3476b53e5aacf6eea4449ed11"
server_key_iv: "7e923b17f1d08d56958ded27a26da507"
server_key_version: "encryptionkey@0"
I20260619 11:09:20.905408 5877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:20.925621 5877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:20.929217 5877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:20.929446 5877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:20.936674 5877 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:33439
I20260619 11:09:20.936703 5944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:33439 every 8 connection(s)
I20260619 11:09:20.937640 5877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:20.938104 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5877
I20260619 11:09:20.938268 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:20.938550 16628 external_mini_cluster.cc:1428] Setting key cc04d0896d4179cf80e544c06e63c73b
I20260619 11:09:20.940828 5945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:20.946071 5945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b: Bootstrap starting.
I20260619 11:09:20.947578 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:20.940032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41497 (local address 127.16.61.62:33439)
0619 11:09:20.940496 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:20.940510 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:20.940545 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:09:20.941231 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:20.941248 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:20.941588 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:20.941926 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:20.942350 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:20.943337 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:20.944332 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:20.944788 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:20.944963 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:20.945916 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:20.945921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:20.945933 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:20.946355 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:20.946407 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:20.946430 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:20.946570 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:20.946754 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 11:09:20.946894 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":143,"threads_started":1}
I20260619 11:09:20.948369 5945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:20.949115 5945 log.cc:824] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:20.950382 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:20.951103 5945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b: No bootstrap required, opened a new log
I20260619 11:09:20.954881 5945 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2e7323ee00473689da2a7295f7a64b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33439 } }
I20260619 11:09:20.955123 5945 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:20.955192 5945 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2e7323ee00473689da2a7295f7a64b, State: Initialized, Role: FOLLOWER
I20260619 11:09:20.955698 5945 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2e7323ee00473689da2a7295f7a64b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33439 } }
I20260619 11:09:20.955860 5945 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:20.955935 5945 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:20.956048 5945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:20.957068 5945 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2e7323ee00473689da2a7295f7a64b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33439 } }
I20260619 11:09:20.957424 5945 leader_election.cc:302] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2e7323ee00473689da2a7295f7a64b; no voters:
I20260619 11:09:20.957739 5945 leader_election.cc:288] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:20.957826 5955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:20.958034 5955 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [term 1 LEADER]: Becoming Leader. State: Replica: cd2e7323ee00473689da2a7295f7a64b, State: Running, Role: LEADER
I20260619 11:09:20.958359 5955 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2e7323ee00473689da2a7295f7a64b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33439 } }
I20260619 11:09:20.958988 5945 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:20.959896 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:20.952371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44783 (local address 127.16.61.62:33439)
0619 11:09:20.952484 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:20.952488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:20.955395 (+ 2907us) server_negotiation.cc:407] Connection header received
0619 11:09:20.955515 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:20.955517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:20.955558 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:20.955629 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:20.956034 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:20.956587 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:20.957732 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:20.958094 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:20.958195 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:20.959238 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:20.959242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:20.959245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:20.959258 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:20.959286 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:20.959298 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:20.959375 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:20.959707 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 11:09:20.959749 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 11:09:20.960402 5957 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2e7323ee00473689da2a7295f7a64b. Latest consensus state: current_term: 1 leader_uuid: "cd2e7323ee00473689da2a7295f7a64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2e7323ee00473689da2a7295f7a64b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33439 } } }
I20260619 11:09:20.960542 5957 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:20.960893 5965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:20.960981 5956 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd2e7323ee00473689da2a7295f7a64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2e7323ee00473689da2a7295f7a64b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 33439 } } }
I20260619 11:09:20.961064 5956 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:20.963991 5965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:20.969223 5965 catalog_manager.cc:1347] Generated new cluster ID: 2203744269914b2682f930d0930f349a
I20260619 11:09:20.969286 5965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:20.989533 5965 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:20.990435 5965 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:21.002992 5965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd2e7323ee00473689da2a7295f7a64b: Generated new TSK 0
I20260619 11:09:21.003959 5965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:21.009052 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5877
W20260619 11:09:21.014480 5862 connection.cc:535] client connection to 127.16.61.62:33439 recv error: Network error: failed to read from TLS socket (remote: 127.16.61.62:33439): Connection reset by peer (error 104)
2026-06-19T11:09:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:21 dist-test-slave-8vcz krb5kdc[5976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:21 dist-test-slave-8vcz krb5kdc[5976](info): set up 2 sockets
Jun 19 11:09:21 dist-test-slave-8vcz krb5kdc[5976](info): commencing operation
krb5kdc: starting...
W20260619 11:09:22.063877 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:22 dist-test-slave-8vcz krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:09:22.180620 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35951
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:35951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:22.289209 5989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:22.289490 5989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:22.289568 5989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:22.293478 5989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:22.293557 5989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:22.293593 5989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:22.293648 5989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:22.293685 5989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:22.298825 5989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:35951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:22.299804 5989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:22.300757 5989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:22.308841 5994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:22.308844 5995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:22.308835 5997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:22.309056 5989 server_base.cc:1034] running on GCE node
I20260619 11:09:22.309659 5989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:22.310702 5989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:22.311908 5989 hybrid_clock.cc:648] HybridClock initialized: now 1781867362311881 us; error 40 us; skew 500 ppm
Jun 19 11:09:22 dist-test-slave-8vcz krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867362, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:22.314586 5989 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:09:22.315378 5989 webserver.cc:460] Webserver started at http://127.16.61.62:43973/ using document root <none> and password file <none>
I20260619 11:09:22.315953 5989 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:22.316040 5989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:22.316255 5989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:22.318123 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "895cd71168fb4244ba50f91d751339d4"
format_stamp: "Formatted at 2026-06-19 11:09:22 on dist-test-slave-8vcz"
server_key: "4dc868152b3ee4db5fe3607ca2adc9a4"
server_key_iv: "4217072540c6cbb3efae89b0187a1cf3"
server_key_version: "encryptionkey@0"
I20260619 11:09:22.318667 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "895cd71168fb4244ba50f91d751339d4"
format_stamp: "Formatted at 2026-06-19 11:09:22 on dist-test-slave-8vcz"
server_key: "4dc868152b3ee4db5fe3607ca2adc9a4"
server_key_iv: "4217072540c6cbb3efae89b0187a1cf3"
server_key_version: "encryptionkey@0"
I20260619 11:09:22.322084 5989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 11:09:22.324368 6004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:22.325513 5989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:22.325744 5989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "895cd71168fb4244ba50f91d751339d4"
format_stamp: "Formatted at 2026-06-19 11:09:22 on dist-test-slave-8vcz"
server_key: "4dc868152b3ee4db5fe3607ca2adc9a4"
server_key_iv: "4217072540c6cbb3efae89b0187a1cf3"
server_key_version: "encryptionkey@0"
I20260619 11:09:22.325865 5989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:22.330252 5989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:22.333302 5989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:22.333484 5989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:22.339989 5989 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:35951
I20260619 11:09:22.340016 6056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:35951 every 8 connection(s)
I20260619 11:09:22.340852 5989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:22.343365 6057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:22.345933 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 5989
I20260619 11:09:22.346067 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:22.346621 16628 external_mini_cluster.cc:1428] Setting key 67e2423f0114cef175c94a568887e38e
I20260619 11:09:22.349308 6057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4: Bootstrap starting.
I20260619 11:09:22.351794 6057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:22.352025 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:22.348186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46431 (local address 127.16.61.62:35951)
0619 11:09:22.348646 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:22.348661 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:22.348694 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:09:22.349300 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:22.349316 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:22.349647 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:22.349981 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:22.350760 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:22.350765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:22.350776 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:22.350786 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:22.350837 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:22.350855 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:22.351009 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:22.351079 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 11:09:22.351248 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":154,"threads_started":1}
I20260619 11:09:22.352677 6057 log.cc:824] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:22.354753 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:22.354811 6057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4: No bootstrap required, opened a new log
I20260619 11:09:22.358459 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:22.356772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55967 (local address 127.16.61.62:35951)
0619 11:09:22.356899 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:22.356903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:22.357100 (+ 197us) server_negotiation.cc:407] Connection header received
0619 11:09:22.357222 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:22.357224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:22.357251 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:22.357313 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:22.358054 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:22.358056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:22.358058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:22.358068 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:22.358089 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:22.358099 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:22.358167 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:22.358272 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 11:09:22.358302 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 11:09:22.359184 6057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895cd71168fb4244ba50f91d751339d4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35951 } }
I20260619 11:09:22.359462 6057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:22.359522 6057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895cd71168fb4244ba50f91d751339d4, State: Initialized, Role: FOLLOWER
I20260619 11:09:22.359983 6057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895cd71168fb4244ba50f91d751339d4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35951 } }
I20260619 11:09:22.360128 6057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:22.360191 6057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:22.360284 6057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:22.361181 6057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895cd71168fb4244ba50f91d751339d4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35951 } }
I20260619 11:09:22.361510 6057 leader_election.cc:302] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 895cd71168fb4244ba50f91d751339d4; no voters:
I20260619 11:09:22.361824 6057 leader_election.cc:288] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:22.361886 6067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:22.362097 6067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [term 1 LEADER]: Becoming Leader. State: Replica: 895cd71168fb4244ba50f91d751339d4, State: Running, Role: LEADER
I20260619 11:09:22.362381 6067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895cd71168fb4244ba50f91d751339d4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35951 } }
I20260619 11:09:22.363075 6057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:22.364137 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 895cd71168fb4244ba50f91d751339d4. Latest consensus state: current_term: 1 leader_uuid: "895cd71168fb4244ba50f91d751339d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895cd71168fb4244ba50f91d751339d4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35951 } } }
I20260619 11:09:22.364259 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:22.364591 6068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "895cd71168fb4244ba50f91d751339d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "895cd71168fb4244ba50f91d751339d4" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35951 } } }
I20260619 11:09:22.364730 6068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:22.364650 6076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:22.368124 6076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:22.374599 6076 catalog_manager.cc:1347] Generated new cluster ID: 3ce6a65fc1fd472aaf3ee1692b21fce6
I20260619 11:09:22.374722 6076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:22.390158 6076 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:22.390522 6076 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:22.399833 6076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 895cd71168fb4244ba50f91d751339d4: Generated new TSK 0
I20260619 11:09:22.400558 6076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:22.403950 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 5989
2026-06-19T11:09:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:22 dist-test-slave-8vcz krb5kdc[6088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:22 dist-test-slave-8vcz krb5kdc[6088](info): set up 2 sockets
Jun 19 11:09:22 dist-test-slave-8vcz krb5kdc[6088](info): commencing operation
krb5kdc: starting...
W20260619 11:09:23.439942 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:23 dist-test-slave-8vcz krb5kdc[6088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:09:23.556216 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:45339
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:39161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:45339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:23.660480 6101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:23.660761 6101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:23.660835 6101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:23.664870 6101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:23.664949 6101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:23.664988 6101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:23.665045 6101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:23.665082 6101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:23.670238 6101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:39161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:45339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:45339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:23.671222 6101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:23.672205 6101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:23.680161 6106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:23.680249 6101 server_base.cc:1034] running on GCE node
W20260619 11:09:23.680220 6109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:23.680162 6107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:23.680963 6101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:23.681957 6101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:23.683137 6101 hybrid_clock.cc:648] HybridClock initialized: now 1781867363683109 us; error 38 us; skew 500 ppm
Jun 19 11:09:23 dist-test-slave-8vcz krb5kdc[6088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867363, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:23.685878 6101 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:09:23.686734 6101 webserver.cc:460] Webserver started at http://127.16.61.62:46701/ using document root <none> and password file <none>
I20260619 11:09:23.687274 6101 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:23.687364 6101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:23.687589 6101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:23.689568 6101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "5d4d4b7e096b4e12aa07770a6ea63290"
format_stamp: "Formatted at 2026-06-19 11:09:23 on dist-test-slave-8vcz"
server_key: "6c0cc7e8c30fcff7ef790f64d0aafdf7"
server_key_iv: "d070d9997e3d4869a21a8ad8e5dc3d1a"
server_key_version: "encryptionkey@0"
I20260619 11:09:23.690124 6101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "5d4d4b7e096b4e12aa07770a6ea63290"
format_stamp: "Formatted at 2026-06-19 11:09:23 on dist-test-slave-8vcz"
server_key: "6c0cc7e8c30fcff7ef790f64d0aafdf7"
server_key_iv: "d070d9997e3d4869a21a8ad8e5dc3d1a"
server_key_version: "encryptionkey@0"
I20260619 11:09:23.693668 6101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:23.695927 6116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:23.697013 6101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:09:23.697145 6101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "5d4d4b7e096b4e12aa07770a6ea63290"
format_stamp: "Formatted at 2026-06-19 11:09:23 on dist-test-slave-8vcz"
server_key: "6c0cc7e8c30fcff7ef790f64d0aafdf7"
server_key_iv: "d070d9997e3d4869a21a8ad8e5dc3d1a"
server_key_version: "encryptionkey@0"
I20260619 11:09:23.697248 6101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:23.715971 6101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:23.718608 6101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:23.718811 6101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:23.726059 6101 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:45339
I20260619 11:09:23.726068 6168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:45339 every 8 connection(s)
I20260619 11:09:23.726980 6101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:23.729645 6169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:23.731490 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6101
I20260619 11:09:23.731631 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:23.731917 16628 external_mini_cluster.cc:1428] Setting key 4626edc2e925e5ddc553254efa80d7dd
I20260619 11:09:23.736083 6169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290: Bootstrap starting.
I20260619 11:09:23.738533 6169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:23.739419 6169 log.cc:824] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:09:23 dist-test-slave-8vcz krb5kdc[6088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:23.741312 6169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290: No bootstrap required, opened a new log
I20260619 11:09:23.745025 6169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4b7e096b4e12aa07770a6ea63290" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45339 } }
I20260619 11:09:23.745301 6169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:23.745391 6169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4d4b7e096b4e12aa07770a6ea63290, State: Initialized, Role: FOLLOWER
I20260619 11:09:23.745822 6169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4b7e096b4e12aa07770a6ea63290" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45339 } }
I20260619 11:09:23.745967 6169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:23.746052 6169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:23.746096 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:23.733157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37563 (local address 127.16.61.62:45339)
0619 11:09:23.733545 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:23.733559 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:23.733593 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:09:23.734207 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:23.734224 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:23.734540 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:23.734798 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:23.735765 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:23.736745 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:23.737949 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:23.738411 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:23.738533 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:23.741346 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:23.741352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:23.741365 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:23.741389 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:23.743107 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:23.743557 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:23.743568 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:23.743572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:23.743730 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:23.744018 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:23.744022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:23.744024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:23.744398 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:23.744615 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:23.744851 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 11:09:23.745137 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":133,"threads_started":1}
I20260619 11:09:23.746147 6169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:23.746984 6169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4b7e096b4e12aa07770a6ea63290" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45339 } }
I20260619 11:09:23.747340 6169 leader_election.cc:302] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d4d4b7e096b4e12aa07770a6ea63290; no voters:
I20260619 11:09:23.747699 6169 leader_election.cc:288] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:23.747807 6174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:23.748063 6174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [term 1 LEADER]: Becoming Leader. State: Replica: 5d4d4b7e096b4e12aa07770a6ea63290, State: Running, Role: LEADER
I20260619 11:09:23.748405 6174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4b7e096b4e12aa07770a6ea63290" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45339 } }
I20260619 11:09:23.748957 6169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:23.749243 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:23.749927 6175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d4d4b7e096b4e12aa07770a6ea63290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4b7e096b4e12aa07770a6ea63290" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45339 } } }
I20260619 11:09:23.750036 6175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:23.750269 6175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d4d4b7e096b4e12aa07770a6ea63290. Latest consensus state: current_term: 1 leader_uuid: "5d4d4b7e096b4e12aa07770a6ea63290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d4d4b7e096b4e12aa07770a6ea63290" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 45339 } } }
I20260619 11:09:23.750341 6175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290 [sys.catalog]: This master's current role is: LEADER
W20260619 11:09:23.753954 6193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:23.754031 6193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:23.754096 6183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:23.754710 6183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:23.760290 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:23.751856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33153 (local address 127.16.61.62:45339)
0619 11:09:23.751972 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:23.751977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:23.752337 (+ 360us) server_negotiation.cc:407] Connection header received
0619 11:09:23.752465 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:23.752469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:23.752510 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:23.752598 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:23.753212 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:23.753718 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:23.754625 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:23.754937 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:23.755057 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:23.756372 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:23.756376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:23.756379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:23.756405 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:23.758037 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:23.758995 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:23.758999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:23.759000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:23.759084 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:23.759418 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:23.759421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:23.759422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:23.759639 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:23.759793 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:23.760039 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 11:09:23.760148 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:09:23.760344 6183 catalog_manager.cc:1347] Generated new cluster ID: 6b67ded071dd4e86ac8c2dc650e58b5a
I20260619 11:09:23.760474 6183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:23.782784 6183 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:23.783712 6183 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:23.791370 6183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d4d4b7e096b4e12aa07770a6ea63290: Generated new TSK 0
I20260619 11:09:23.792061 6183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:23.800915 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6101
2026-06-19T11:09:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:23 dist-test-slave-8vcz krb5kdc[6200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:23 dist-test-slave-8vcz krb5kdc[6200](info): set up 2 sockets
Jun 19 11:09:23 dist-test-slave-8vcz krb5kdc[6200](info): commencing operation
krb5kdc: starting...
W20260619 11:09:24.824061 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:24 dist-test-slave-8vcz krb5kdc[6200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:09:24.941943 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:37757
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:42631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:37757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:25.048277 6213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:25.048593 6213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:25.048673 6213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:25.052598 6213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:25.052685 6213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:25.052723 6213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:25.052779 6213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:25.052817 6213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:25.057875 6213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:42631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:37757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:37757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:25.058859 6213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:25.059834 6213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:25.068005 6218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:25.068041 6219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:25.068030 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:25.068187 6213 server_base.cc:1034] running on GCE node
I20260619 11:09:25.069083 6213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:25.070173 6213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:25.071349 6213 hybrid_clock.cc:648] HybridClock initialized: now 1781867365071334 us; error 24 us; skew 500 ppm
Jun 19 11:09:25 dist-test-slave-8vcz krb5kdc[6200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867365, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:25.076949 6213 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:09:25.077867 6213 webserver.cc:460] Webserver started at http://127.16.61.62:39315/ using document root <none> and password file <none>
I20260619 11:09:25.078394 6213 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:25.078454 6213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:25.078670 6213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:25.080611 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "8430523815824867978c4743cc147d0d"
format_stamp: "Formatted at 2026-06-19 11:09:25 on dist-test-slave-8vcz"
server_key: "12f66d3c5d0c74dd46dd6fd0c7bdccd4"
server_key_iv: "4beb0badad59ef1450121c96e879f666"
server_key_version: "encryptionkey@0"
I20260619 11:09:25.081143 6213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "8430523815824867978c4743cc147d0d"
format_stamp: "Formatted at 2026-06-19 11:09:25 on dist-test-slave-8vcz"
server_key: "12f66d3c5d0c74dd46dd6fd0c7bdccd4"
server_key_iv: "4beb0badad59ef1450121c96e879f666"
server_key_version: "encryptionkey@0"
I20260619 11:09:25.084626 6213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:09:25.086812 6228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:25.088027 6213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:09:25.088156 6213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "8430523815824867978c4743cc147d0d"
format_stamp: "Formatted at 2026-06-19 11:09:25 on dist-test-slave-8vcz"
server_key: "12f66d3c5d0c74dd46dd6fd0c7bdccd4"
server_key_iv: "4beb0badad59ef1450121c96e879f666"
server_key_version: "encryptionkey@0"
I20260619 11:09:25.088259 6213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:25.099308 6213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:25.101954 6213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:25.102141 6213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:25.109174 6213 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:37757
I20260619 11:09:25.109197 6280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:37757 every 8 connection(s)
I20260619 11:09:25.110044 6213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:25.112998 6281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:25.117288 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6213
I20260619 11:09:25.117434 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:25.117723 16628 external_mini_cluster.cc:1428] Setting key 38dc471677265ef76cf745faed97e6fe
I20260619 11:09:25.118451 6281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d: Bootstrap starting.
I20260619 11:09:25.120942 6281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:25.121788 6281 log.cc:824] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:25.123718 6281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d: No bootstrap required, opened a new log
I20260619 11:09:25.127588 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:25.119061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57949 (local address 127.16.61.62:37757)
0619 11:09:25.119476 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:25.119498 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:25.119540 (+ 42us) server_negotiation.cc:407] Connection header received
0619 11:09:25.120355 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:25.120376 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:25.120773 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:25.121147 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:25.121549 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:25.122896 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:25.123978 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:25.124631 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:25.124788 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:25.125562 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:25.125571 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:25.125587 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:25.126082 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:25.126147 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:25.126176 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:25.126270 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:25.126384 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 11:09:25.126589 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":107,"threads_started":1}
I20260619 11:09:25.128293 6281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8430523815824867978c4743cc147d0d" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37757 } }
I20260619 11:09:25.128531 6281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:25.128592 6281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8430523815824867978c4743cc147d0d, State: Initialized, Role: FOLLOWER
I20260619 11:09:25.128980 6281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8430523815824867978c4743cc147d0d" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37757 } }
I20260619 11:09:25.129127 6281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:25.129211 6281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:25.129307 6281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:25.130295 6281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8430523815824867978c4743cc147d0d" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37757 } }
I20260619 11:09:25.130533 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:25.130677 6281 leader_election.cc:302] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8430523815824867978c4743cc147d0d; no voters:
I20260619 11:09:25.130980 6281 leader_election.cc:288] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:25.131091 6286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:25.131345 6286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [term 1 LEADER]: Becoming Leader. State: Replica: 8430523815824867978c4743cc147d0d, State: Running, Role: LEADER
I20260619 11:09:25.131744 6286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8430523815824867978c4743cc147d0d" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37757 } }
I20260619 11:09:25.132287 6281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:25.133010 6286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8430523815824867978c4743cc147d0d. Latest consensus state: current_term: 1 leader_uuid: "8430523815824867978c4743cc147d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8430523815824867978c4743cc147d0d" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37757 } } }
I20260619 11:09:25.133190 6286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:25.133436 6287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8430523815824867978c4743cc147d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8430523815824867978c4743cc147d0d" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 37757 } } }
I20260619 11:09:25.133553 6287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:25.133713 6300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:25.136962 6300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:25.137944 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:25.132770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39055 (local address 127.16.61.62:37757)
0619 11:09:25.132868 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:25.132872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:25.133206 (+ 334us) server_negotiation.cc:407] Connection header received
0619 11:09:25.133448 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:25.133452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:25.133505 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:25.133560 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:25.133989 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:25.134703 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:25.135850 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:25.136166 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:25.136240 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:25.137475 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:25.137480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:25.137483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:25.137495 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:25.137522 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:25.137535 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:25.137580 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:25.137761 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 11:09:25.137816 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 11:09:25.142993 6300 catalog_manager.cc:1347] Generated new cluster ID: 83c13eef437141a99a7066b0712ec106
I20260619 11:09:25.143065 6300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:25.156872 6300 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:25.158068 6300 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:25.163705 6300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8430523815824867978c4743cc147d0d: Generated new TSK 0
I20260619 11:09:25.164345 6300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:25.174839 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6213
2026-06-19T11:09:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:25 dist-test-slave-8vcz krb5kdc[6312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:25 dist-test-slave-8vcz krb5kdc[6312](info): set up 2 sockets
Jun 19 11:09:25 dist-test-slave-8vcz krb5kdc[6312](info): commencing operation
krb5kdc: starting...
W20260619 11:09:26.199942 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:26 dist-test-slave-8vcz krb5kdc[6312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:09:26.318853 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:36759
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:36759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:26.428570 6325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:26.428826 6325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:26.428867 6325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:26.432821 6325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:26.432875 6325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:26.432894 6325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:26.432910 6325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:26.432955 6325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:26.438057 6325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:36759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:36759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:26.439023 6325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:26.440006 6325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:26.447961 6330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:26.447969 6331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:26.448077 6325 server_base.cc:1034] running on GCE node
W20260619 11:09:26.448168 6333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:26.448680 6325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:26.449702 6325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:26.450881 6325 hybrid_clock.cc:648] HybridClock initialized: now 1781867366450863 us; error 30 us; skew 500 ppm
Jun 19 11:09:26 dist-test-slave-8vcz krb5kdc[6312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867366, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:26.453472 6325 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:09:26.454313 6325 webserver.cc:460] Webserver started at http://127.16.61.62:34413/ using document root <none> and password file <none>
I20260619 11:09:26.454838 6325 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:26.454926 6325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:26.455147 6325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:26.457086 6325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "bae742dec5c541e09a0af5c41137ba17"
format_stamp: "Formatted at 2026-06-19 11:09:26 on dist-test-slave-8vcz"
server_key: "2828e1ddc4eb2b8bbff0a5611b02b5f5"
server_key_iv: "304caf45c8af6f5064c74ddcee37d7f3"
server_key_version: "encryptionkey@0"
I20260619 11:09:26.457628 6325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "bae742dec5c541e09a0af5c41137ba17"
format_stamp: "Formatted at 2026-06-19 11:09:26 on dist-test-slave-8vcz"
server_key: "2828e1ddc4eb2b8bbff0a5611b02b5f5"
server_key_iv: "304caf45c8af6f5064c74ddcee37d7f3"
server_key_version: "encryptionkey@0"
I20260619 11:09:26.461076 6325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:26.463430 6340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:26.464622 6325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:26.464753 6325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "bae742dec5c541e09a0af5c41137ba17"
format_stamp: "Formatted at 2026-06-19 11:09:26 on dist-test-slave-8vcz"
server_key: "2828e1ddc4eb2b8bbff0a5611b02b5f5"
server_key_iv: "304caf45c8af6f5064c74ddcee37d7f3"
server_key_version: "encryptionkey@0"
I20260619 11:09:26.464856 6325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:26.469049 6325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:26.471750 6325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:26.471921 6325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:26.478986 6325 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:36759
I20260619 11:09:26.479002 6392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:36759 every 8 connection(s)
I20260619 11:09:26.479882 6325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:26.482511 6393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:26.484469 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6325
I20260619 11:09:26.484614 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:26.484886 16628 external_mini_cluster.cc:1428] Setting key 0202cbf7eec101a195da8f4b31289fdf
I20260619 11:09:26.488669 6393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17: Bootstrap starting.
I20260619 11:09:26.490342 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:26.486197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60053 (local address 127.16.61.62:36759)
0619 11:09:26.486590 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:26.486609 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:26.486652 (+ 43us) server_negotiation.cc:407] Connection header received
0619 11:09:26.487588 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:26.487609 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:26.487982 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:26.488349 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:26.489024 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:26.489029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:26.489043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:26.489055 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:26.489112 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:26.489136 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:26.489269 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:26.489342 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 11:09:26.489475 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":101,"threads_started":1}
I20260619 11:09:26.491201 6393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:26.492026 6393 log.cc:824] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:26.493085 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:26.494177 6393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17: No bootstrap required, opened a new log
I20260619 11:09:26.496754 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:26.495236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40781 (local address 127.16.61.62:36759)
0619 11:09:26.495354 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:26.495359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:26.495516 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:09:26.495589 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:26.495592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:26.495623 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:26.495749 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:26.496412 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:26.496416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:26.496418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 11:09:26.496429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:26.496456 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 11:09:26.496468 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:26.496556 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:26.496613 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 11:09:26.496645 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 11:09:26.498327 6393 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae742dec5c541e09a0af5c41137ba17" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36759 } }
I20260619 11:09:26.498530 6393 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 11:09:26.498611 6393 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae742dec5c541e09a0af5c41137ba17, State: Initialized, Role: FOLLOWER
I20260619 11:09:26.499002 6393 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae742dec5c541e09a0af5c41137ba17" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36759 } }
I20260619 11:09:26.499135 6393 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 11:09:26.499202 6393 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 11:09:26.499312 6393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:26.500197 6393 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae742dec5c541e09a0af5c41137ba17" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36759 } }
I20260619 11:09:26.500521 6393 leader_election.cc:302] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae742dec5c541e09a0af5c41137ba17; no voters:
I20260619 11:09:26.500830 6393 leader_election.cc:288] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 11:09:26.500994 6403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:26.501240 6403 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [term 1 LEADER]: Becoming Leader. State: Replica: bae742dec5c541e09a0af5c41137ba17, State: Running, Role: LEADER
I20260619 11:09:26.501591 6403 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae742dec5c541e09a0af5c41137ba17" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36759 } }
I20260619 11:09:26.502029 6393 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:26.503302 6404 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae742dec5c541e09a0af5c41137ba17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae742dec5c541e09a0af5c41137ba17" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36759 } } }
I20260619 11:09:26.503331 6405 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae742dec5c541e09a0af5c41137ba17. Latest consensus state: current_term: 1 leader_uuid: "bae742dec5c541e09a0af5c41137ba17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae742dec5c541e09a0af5c41137ba17" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 36759 } } }
I20260619 11:09:26.503423 6405 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:26.503422 6404 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:26.503865 6412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:26.506884 6412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:26.512323 6412 catalog_manager.cc:1347] Generated new cluster ID: ebfbdfb6d4234ff1997f0ed983b8a7e9
I20260619 11:09:26.512385 6412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:26.521556 6412 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:26.521826 6412 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:26.526590 6412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae742dec5c541e09a0af5c41137ba17: Generated new TSK 0
I20260619 11:09:26.527143 6412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 11:09:26.533033 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6325
2026-06-19T11:09:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1357 ms)
[----------] 18 tests from AuthTokenIssuingTest (24992 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:26 dist-test-slave-8vcz krb5kdc[6424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:26 dist-test-slave-8vcz krb5kdc[6424](info): set up 2 sockets
Jun 19 11:09:26 dist-test-slave-8vcz krb5kdc[6424](info): commencing operation
krb5kdc: starting...
W20260619 11:09:27.563915 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:27 dist-test-slave-8vcz krb5kdc[6424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:09:27.680620 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:38499
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:38499,127.16.61.61:44773,127.16.61.60:41879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:27.784842 6437 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:09:27.785135 6437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:27.785207 6437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:27.785285 6437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:27.789139 6437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:27.789215 6437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:27.789251 6437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:27.789302 6437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:27.789335 6437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:27.794358 6437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:38499,127.16.61.61:44773,127.16.61.60:41879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:38499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:27.795286 6437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:27.796226 6437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:27.804199 6445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:27.804313 6442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:27.804175 6443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:27.804488 6437 server_base.cc:1034] running on GCE node
I20260619 11:09:27.805058 6437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:27.806041 6437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:27.807205 6437 hybrid_clock.cc:648] HybridClock initialized: now 1781867367807176 us; error 40 us; skew 500 ppm
Jun 19 11:09:27 dist-test-slave-8vcz krb5kdc[6424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867367, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:27.809906 6437 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:09:27.810726 6437 webserver.cc:460] Webserver started at http://127.16.61.62:41039/ using document root <none> and password file <none>
I20260619 11:09:27.811244 6437 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:27.811344 6437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:27.811569 6437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:27.813478 6437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "7498de29f74a4c93812ec1cd40664a68"
format_stamp: "Formatted at 2026-06-19 11:09:27 on dist-test-slave-8vcz"
server_key: "17083b14740f8b425ba49bcda81f10dd"
server_key_iv: "5b71e1ee20f2db0b4c17942308d22c8a"
server_key_version: "encryptionkey@0"
I20260619 11:09:27.814023 6437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "7498de29f74a4c93812ec1cd40664a68"
format_stamp: "Formatted at 2026-06-19 11:09:27 on dist-test-slave-8vcz"
server_key: "17083b14740f8b425ba49bcda81f10dd"
server_key_iv: "5b71e1ee20f2db0b4c17942308d22c8a"
server_key_version: "encryptionkey@0"
I20260619 11:09:27.817535 6437 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:27.819851 6452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:27.821026 6437 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:27.821164 6437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "7498de29f74a4c93812ec1cd40664a68"
format_stamp: "Formatted at 2026-06-19 11:09:27 on dist-test-slave-8vcz"
server_key: "17083b14740f8b425ba49bcda81f10dd"
server_key_iv: "5b71e1ee20f2db0b4c17942308d22c8a"
server_key_version: "encryptionkey@0"
I20260619 11:09:27.821276 6437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:27.842765 6437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:27.845913 6437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:27.846112 6437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:27.853017 6437 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:38499
I20260619 11:09:27.853302 6504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:38499 every 8 connection(s)
I20260619 11:09:27.853996 6437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:27.855573 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6437
I20260619 11:09:27.855746 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:27.856032 16628 external_mini_cluster.cc:1428] Setting key 3d22113e5e25a168718eb1e782353af7
I20260619 11:09:27.856739 6505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:27.861639 6505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.61.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.61@KRBTEST.COM" created.
Jun 19 11:09:27 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867367, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:27.877892 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:27.864380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.16.61.62:38499)
0619 11:09:27.864892 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:27.864909 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:27.865032 (+ 123us) server_negotiation.cc:407] Connection header received
0619 11:09:27.865987 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:27.865991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:27.866159 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:27.866378 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:27.867150 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:27.868583 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:27.870034 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:27.870392 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:27.870506 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:27.873152 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:27.873157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:27.873165 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:27.873184 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:27.874773 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:27.875256 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:27.875262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:27.875264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:27.875417 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:27.875815 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:27.875820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:27.875822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:27.876251 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:27.876403 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:27.876858 (+ 455us) server_negotiation.cc:299] Negotiation successful
0619 11:09:27.877074 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":396,"spinlock_wait_cycles":51584,"thread_start_us":148,"threads_started":1}
I20260619 11:09:27.877890 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:27.864380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:38499 (local address 127.0.0.1:45054)
0619 11:09:27.864892 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:27.864931 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:27.865724 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:27.866373 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:27.866377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:27.866683 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:27.866975 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:27.866996 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:27.868737 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:27.868743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:27.869927 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:27.869937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:27.870545 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:27.870552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:27.870714 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:27.871263 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:27.871303 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:27.872998 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:27.874904 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:27.874914 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:27.874922 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:27.875145 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:27.875538 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:27.875541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:27.875543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:27.875711 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:27.876376 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:27.876382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:27.876559 (+ 177us) client_negotiation.cc:769] Sending connection context
0619 11:09:27.876765 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 11:09:27.877136 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":70,"thread_start_us":148,"threads_started":1}
I20260619 11:09:27.880268 6505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:27.880959 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:27.880686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54320)
0619 11:09:27.880793 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:27.880893 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 11:09:27.881168 6455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
I20260619 11:09:27.882020 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:27.881832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54332)
0619 11:09:27.881932 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:27.881973 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:09:27.882180 6505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:44773: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab.
Entry for principal kudu/127.16.61.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab.
I20260619 11:09:27.883020 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.61:44773
--webserver_interface=127.16.61.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab
--principal=kudu/127.16.61.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:38499,127.16.61.61:44773,127.16.61.60:41879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:09:27.931422 6505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } attempt: 1
I20260619 11:09:27.932384 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:27.932101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54336)
0619 11:09:27.932251 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:27.932310 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 11:09:27.933104 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:27.932892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54350)
0619 11:09:27.933001 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:27.933056 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:09:27.933228 6505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:44773: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
W20260619 11:09:27.988698 6512 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:09:27.988994 6512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:27.989065 6512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:27.989146 6512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:27.993037 6512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:27.993119 6512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:27.993158 6512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:27.993193 6512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:27.993234 6512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:27.998247 6512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:38499,127.16.61.61:44773,127.16.61.60:41879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab
--principal=kudu/127.16.61.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.61:44773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.61.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:27.999205 6512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:28.000159 6512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:09:28.001387 6505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } attempt: 2
I20260619 11:09:28.002103 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.001846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54352)
0619 11:09:28.001972 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.002036 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 11:09:28.002768 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.002565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54356)
0619 11:09:28.002668 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.002715 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 11:09:28.002907 6505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:44773: Network error: Client connection negotiation failed: client connection to 127.16.61.61:44773: connect: Connection refused (error 111)
W20260619 11:09:28.007978 6517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:28.007984 6518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:28.007987 6520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:28.008496 6512 server_base.cc:1034] running on GCE node
I20260619 11:09:28.009066 6512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:28.010066 6512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:28.011225 6512 hybrid_clock.cc:648] HybridClock initialized: now 1781867368011207 us; error 28 us; skew 500 ppm
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867368, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:28.013851 6512 init.cc:377] Logged in from keytab as kudu/127.16.61.61@KRBTEST.COM (short username kudu)
I20260619 11:09:28.014647 6512 webserver.cc:460] Webserver started at http://127.16.61.61:35211/ using document root <none> and password file <none>
I20260619 11:09:28.015156 6512 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:28.015213 6512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:28.015450 6512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:28.017370 6512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/data/instance:
uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc"
format_stamp: "Formatted at 2026-06-19 11:09:28 on dist-test-slave-8vcz"
server_key: "40385b563cb271e21ec51c5480b81742"
server_key_iv: "9e5e1fcdf190c72e67553946bad80934"
server_key_version: "encryptionkey@0"
I20260619 11:09:28.017935 6512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/wal/instance:
uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc"
format_stamp: "Formatted at 2026-06-19 11:09:28 on dist-test-slave-8vcz"
server_key: "40385b563cb271e21ec51c5480b81742"
server_key_iv: "9e5e1fcdf190c72e67553946bad80934"
server_key_version: "encryptionkey@0"
I20260619 11:09:28.021359 6512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 11:09:28.023536 6527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:28.024724 6512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:28.024858 6512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/wal
uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc"
format_stamp: "Formatted at 2026-06-19 11:09:28 on dist-test-slave-8vcz"
server_key: "40385b563cb271e21ec51c5480b81742"
server_key_iv: "9e5e1fcdf190c72e67553946bad80934"
server_key_version: "encryptionkey@0"
I20260619 11:09:28.024962 6512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:28.046140 6512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:28.048758 6512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:28.048897 6512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:28.055622 6512 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.61:44773
I20260619 11:09:28.056062 6579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.61:44773 every 8 connection(s)
I20260619 11:09:28.056623 6512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
I20260619 11:09:28.058626 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6512
I20260619 11:09:28.058758 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-1/wal/instance
I20260619 11:09:28.059041 16628 external_mini_cluster.cc:1428] Setting key 6a12717c16985bc834ef367eaa923d68
I20260619 11:09:28.059531 6580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:28.063413 6580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.61.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.60@KRBTEST.COM" created.
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867368, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:28.078146 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.065218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.16.61.62:38499)
0619 11:09:28.065380 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.065386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.066137 (+ 751us) server_negotiation.cc:407] Connection header received
0619 11:09:28.067066 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.067069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.067119 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.067204 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.068552 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.069272 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.070826 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.071245 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.071362 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.073827 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.073831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.073834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.073867 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.076002 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.076498 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.076501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.076503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.076610 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.076957 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.076961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.076963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.077197 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.077310 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.077848 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.077977 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 11:09:28.079026 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.065528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:38499 (local address 127.0.0.1:45058)
0619 11:09:28.065974 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.066022 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:28.066873 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:28.067465 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:28.067472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:28.067838 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:28.068387 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.068400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.069426 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.069432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.070680 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.070690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.071406 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.071408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.071541 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.072161 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:28.072178 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:28.073656 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:28.076119 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.076124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.076138 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.076390 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.076707 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.076711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.076713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.076855 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.077308 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:28.077315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:28.077597 (+ 282us) client_negotiation.cc:769] Sending connection context
0619 11:09:28.077790 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 11:09:28.078091 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":138,"threads_started":1}
I20260619 11:09:28.080415 6580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab.
Entry for principal kudu/127.16.61.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab.
I20260619 11:09:28.082777 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.60:41879
--webserver_interface=127.16.61.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab
--principal=kudu/127.16.61.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:38499,127.16.61.61:44773,127.16.61.60:41879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867368, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
I20260619 11:09:28.089421 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.080876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54362)
0619 11:09:28.080990 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.081003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:28.081069 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:28.081687 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:28.081690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:28.081886 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:28.081981 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.081986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.083084 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.083089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.083585 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.083592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.084298 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.084303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.084420 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.084833 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:28.084851 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:28.086005 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:28.087956 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.087960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.087962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.088147 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.088428 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.088433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.088435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.088513 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.088932 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:28.088935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:28.089013 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 11:09:28.089132 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 11:09:28.089247 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 11:09:28.089466 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.080996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.16.61.61:44773)
0619 11:09:28.081285 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.081297 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.081321 (+ 24us) server_negotiation.cc:407] Connection header received
0619 11:09:28.081376 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.081384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.081572 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.081682 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.082111 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.082948 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.083718 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.084178 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.084267 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.086139 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.086155 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.086165 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.086183 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.087835 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.088251 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.088257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.088258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.088337 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.088606 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.088609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.088610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.088821 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.088966 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.089206 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.089323 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":136,"threads_started":1}
I20260619 11:09:28.091117 6580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:28.091872 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.091602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41166)
0619 11:09:28.091754 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.091818 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 11:09:28.092052 6531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
I20260619 11:09:28.092969 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.092741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41174)
0619 11:09:28.092867 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.092922 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 11:09:28.093130 6580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:41879: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
I20260619 11:09:28.094059 6505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } attempt: 3
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867367, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
I20260619 11:09:28.102564 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.094429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54368)
0619 11:09:28.094594 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.094606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:28.094703 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:28.095053 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:28.095056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:28.095226 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:28.095321 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.095327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.096355 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.096358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.096949 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.096958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.097548 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.097551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.097632 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.098079 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:28.098090 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:28.099240 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:28.101154 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.101157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.101158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.101373 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.101668 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.101671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.101672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.101769 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.102177 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:28.102179 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:28.102257 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 11:09:28.102346 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 11:09:28.102452 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260619 11:09:28.102670 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.094496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.16.61.61:44773)
0619 11:09:28.094603 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.094606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.094644 (+ 38us) server_negotiation.cc:407] Connection header received
0619 11:09:28.094779 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.094784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.094914 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.095041 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.095446 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.096244 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.097059 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.097437 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.097533 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.099375 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.099381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.099387 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.099419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.101046 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.101463 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.101466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.101468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.101556 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.101864 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.101867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.101869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.102075 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.102172 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.102414 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.102531 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 11:09:28.103643 6505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:28.104321 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.104068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41190)
0619 11:09:28.104186 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.104241 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 11:09:28.105057 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.104841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41198)
0619 11:09:28.104974 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.105014 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 11:09:28.105340 6505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:41879: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
I20260619 11:09:28.136528 6505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } attempt: 1
I20260619 11:09:28.137425 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.137077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41202)
0619 11:09:28.137250 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.137345 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 11:09:28.138195 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.137970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41210)
0619 11:09:28.138089 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.138143 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 11:09:28.138337 6505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:41879: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
I20260619 11:09:28.142354 6580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } attempt: 1
I20260619 11:09:28.143019 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.142744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41216)
0619 11:09:28.142880 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.142943 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:09:28.143766 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.143525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41232)
0619 11:09:28.143631 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.143704 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:09:28.143901 6580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:41879: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
W20260619 11:09:28.194831 6586 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:09:28.195082 6586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:28.195132 6586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:28.195221 6586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:28.199182 6586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:28.199235 6586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:28.199254 6586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:28.199271 6586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:28.199326 6586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:28.204751 6586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:38499,127.16.61.61:44773,127.16.61.60:41879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab
--principal=kudu/127.16.61.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.60:41879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.61.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:28.205698 6586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:28.206634 6586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 11:09:28.212071 6580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } attempt: 2
I20260619 11:09:28.212821 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.212561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41238)
0619 11:09:28.212697 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.212752 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 11:09:28.213490 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.213279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41250)
0619 11:09:28.213381 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.213427 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 11:09:28.213497 6505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } attempt: 2
W20260619 11:09:28.213647 6580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:41879: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
I20260619 11:09:28.214181 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.213908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41262)
0619 11:09:28.214038 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.214101 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 11:09:28.215008 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.214689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41276)
0619 11:09:28.214861 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.214913 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260619 11:09:28.215126 6505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:41879: Network error: Client connection negotiation failed: client connection to 127.16.61.60:41879: connect: Connection refused (error 111)
W20260619 11:09:28.215490 6593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:28.215631 6592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:28.215458 6595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:28.215793 6586 server_base.cc:1034] running on GCE node
I20260619 11:09:28.216318 6586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:28.217278 6586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:28.218462 6586 hybrid_clock.cc:648] HybridClock initialized: now 1781867368218454 us; error 25 us; skew 500 ppm
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867368, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:28.221042 6586 init.cc:377] Logged in from keytab as kudu/127.16.61.60@KRBTEST.COM (short username kudu)
I20260619 11:09:28.221869 6586 webserver.cc:460] Webserver started at http://127.16.61.60:39017/ using document root <none> and password file <none>
I20260619 11:09:28.222400 6586 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:28.222489 6586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:28.222703 6586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:28.224687 6586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/data/instance:
uuid: "2dd6663674884be58503e60dbd7cfb71"
format_stamp: "Formatted at 2026-06-19 11:09:28 on dist-test-slave-8vcz"
server_key: "e4b2d7ebf22f09498a9dd5a8a03880d0"
server_key_iv: "2f90eb3b90079a11180a4e01c7893be2"
server_key_version: "encryptionkey@0"
I20260619 11:09:28.225220 6586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/wal/instance:
uuid: "2dd6663674884be58503e60dbd7cfb71"
format_stamp: "Formatted at 2026-06-19 11:09:28 on dist-test-slave-8vcz"
server_key: "e4b2d7ebf22f09498a9dd5a8a03880d0"
server_key_iv: "2f90eb3b90079a11180a4e01c7893be2"
server_key_version: "encryptionkey@0"
I20260619 11:09:28.228807 6586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 11:09:28.231083 6602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:28.232301 6586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 11:09:28.232432 6586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/wal
uuid: "2dd6663674884be58503e60dbd7cfb71"
format_stamp: "Formatted at 2026-06-19 11:09:28 on dist-test-slave-8vcz"
server_key: "e4b2d7ebf22f09498a9dd5a8a03880d0"
server_key_iv: "2f90eb3b90079a11180a4e01c7893be2"
server_key_version: "encryptionkey@0"
I20260619 11:09:28.232542 6586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:28.250973 6586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:28.253764 6586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:28.253957 6586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:28.261575 6586 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.60:41879
I20260619 11:09:28.261843 6654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.60:41879 every 8 connection(s)
I20260619 11:09:28.262581 6586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
I20260619 11:09:28.265208 6655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:28.268808 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6586
I20260619 11:09:28.268940 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781867094678158-16628-0/minicluster-data/master-2/wal/instance
I20260619 11:09:28.268709 6655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:28.269248 16628 external_mini_cluster.cc:1428] Setting key ce98fdc1d8052363a0b7ff828a12aafa
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867368, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
I20260619 11:09:28.281714 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.271305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.16.61.62:38499)
0619 11:09:28.271443 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.271447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.271647 (+ 200us) server_negotiation.cc:407] Connection header received
0619 11:09:28.272130 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.272133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.272177 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.272248 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.273151 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.273733 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.274847 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.275204 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.275302 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.278066 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.278069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.278071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.278092 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.279615 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.280165 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.280169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.280171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.280249 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.280590 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.280595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.280596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.280792 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.280895 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.281395 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.281510 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 11:09:28.282462 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.270596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.16.61.60:41879)
0619 11:09:28.270951 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.270964 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.270994 (+ 30us) server_negotiation.cc:407] Connection header received
0619 11:09:28.271921 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.271950 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.272172 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.272359 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.273067 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.274265 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.275242 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.275757 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.275864 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.278397 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.278403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.278415 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.278448 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.279990 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.280404 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.280409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.280411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.280496 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.280775 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.280780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.280782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.281150 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.281352 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.281551 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.281691 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260619 11:09:28.282460 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.271129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:38499 (local address 127.0.0.1:45074)
0619 11:09:28.271503 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.271553 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:28.271931 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:28.272317 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:28.272322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:28.272758 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:28.272982 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.273024 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.273832 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.273836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.274739 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.274747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.275332 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.275336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.275531 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.276224 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:28.276283 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:28.277885 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:28.279782 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.279790 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.279799 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.280064 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.280350 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.280355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.280357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.280490 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.280906 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:28.280914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:28.281163 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 11:09:28.281327 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 11:09:28.281627 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":22,"spinlock_wait_cycles":1408,"thread_start_us":85,"threads_started":1}
I20260619 11:09:28.284183 6655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867368, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:28.292663 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.284671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:44773 (local address 127.0.0.1:54376)
0619 11:09:28.284987 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.285001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:28.285078 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:28.285337 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:28.285340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:28.285463 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:28.285555 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.285560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.286304 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.286310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.286836 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.286844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.287396 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.287401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.287500 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.288030 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:28.288041 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:28.289333 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:28.291190 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.291195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.291196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.291395 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.291716 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.291719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.291720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.291790 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.292172 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:28.292175 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:28.292279 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 11:09:28.292386 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 11:09:28.292504 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260619 11:09:28.292683 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.284761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.16.61.61:44773)
0619 11:09:28.284910 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.284915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.285045 (+ 130us) server_negotiation.cc:407] Connection header received
0619 11:09:28.285175 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.285178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.285227 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.285317 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.285666 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.286201 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.286949 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.287193 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.287264 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.289468 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.289472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.289477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.289506 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.291080 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.291495 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.291501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.291504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.291583 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.291889 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.291892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.291894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.292064 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.292182 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.292439 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.292545 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 11:09:28.293336 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.284864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.16.61.62:38499)
0619 11:09:28.284973 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.284976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.284989 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:09:28.285042 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.285047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.285086 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.285163 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.285807 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.286282 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.287031 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.287344 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.287445 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.289968 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.289971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.289974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.289999 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.291459 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.291950 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.291954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.291956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.292029 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.292314 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.292319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.292321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.292481 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.292696 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.293096 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.293204 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:09:28.293758 6655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867368, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
I20260619 11:09:28.301913 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.294206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41286)
0619 11:09:28.294307 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.294322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:28.294395 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:28.294615 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:28.294619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:28.294779 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:28.294870 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.294876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.295873 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.295877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.296545 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.296553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.297120 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.297124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.297205 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.297603 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:28.297617 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:28.298742 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:28.300565 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.300571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.300574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.300751 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.301019 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.301024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.301026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.301100 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.301438 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:28.301441 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:28.301517 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 11:09:28.301620 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 11:09:28.301749 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 11:09:28.301937 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.294248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.16.61.60:41879)
0619 11:09:28.294349 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.294353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.294366 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:09:28.294460 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.294464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.294506 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.294586 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.294987 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.295782 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.296646 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.297006 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.297114 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.298897 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.298907 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.298910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.298936 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.300464 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.300842 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.300848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.300850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.300933 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.301188 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.301191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.301193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.301344 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.301445 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.301685 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.301797 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":43}
I20260619 11:09:28.303290 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.294569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.16.61.61:44773)
0619 11:09:28.294684 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.294688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.294699 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:09:28.294732 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.294734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.294774 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.294847 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.295546 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.296240 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.297110 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.297440 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.297560 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.300208 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.300211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.300212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.300231 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.301582 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.301999 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.302004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.302005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.302263 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.302533 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.302538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.302540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.302697 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.302920 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.303038 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.303144 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:09:28.304780 6580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } attempt: 3
I20260619 11:09:28.305655 6655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71: Bootstrap starting.
I20260619 11:09:28.306888 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:28.308450 6655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:28.309238 6655 log.cc:824] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867368, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
I20260619 11:09:28.308193 6479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7498de29f74a4c93812ec1cd40664a68"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45088
I20260619 11:09:28.311416 6655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71: No bootstrap required, opened a new log
I20260619 11:09:28.313285 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.305112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41298)
0619 11:09:28.305255 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.305269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:28.305346 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:28.305578 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:28.305581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:28.305719 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:28.305785 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.305790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.306636 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.306641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.307292 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.307301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.307776 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.307779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.307859 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.308302 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:28.308316 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:28.309742 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:28.311694 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.311697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.311699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.311971 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.312259 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.312262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.312264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.312357 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.312782 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:28.312785 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:28.312878 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 11:09:28.312997 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 11:09:28.313130 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 11:09:28.313305 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.305163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.16.61.60:41879)
0619 11:09:28.305271 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.305276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.305310 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:09:28.305417 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.305423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.305465 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.305894 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.305990 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.306536 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.307408 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.307629 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.307777 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.309884 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.309888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.309892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.309928 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.311564 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.312082 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.312085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.312087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.312166 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.312451 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.312453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.312455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.312680 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.312774 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.313056 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.313163 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 11:09:28.314285 6505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } attempt: 3
I20260619 11:09:28.315220 6655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } }
I20260619 11:09:28.315392 6655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd6663674884be58503e60dbd7cfb71, State: Initialized, Role: FOLLOWER
I20260619 11:09:28.315641 6655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } }
I20260619 11:09:28.316109 6661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.316231 6661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:28.316669 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc: Bootstrap starting.
I20260619 11:09:28.316840 6655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:28.319274 6580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:28.320182 6580 log.cc:824] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc: Log is configured to *not* fsync() on all Append() calls
Jun 19 11:09:28 dist-test-slave-8vcz krb5kdc[6424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867367, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
I20260619 11:09:28.322288 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc: No bootstrap required, opened a new log
W20260619 11:09:28.323529 6672 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:28.323596 6672 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:28.320576 6586 master_runner.cc:186] Error getting master registration for 127.16.61.62:38499: OK, instance_id { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" instance_seqno: 1781867367850929 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:09:28.324335 6586 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:09:28.326139 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.314687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:41879 (local address 127.0.0.1:41302)
0619 11:09:28.314850 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:28.314863 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:28.314945 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:28.315563 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:28.315566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:28.315776 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:28.315876 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.315883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.317800 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.317806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.318579 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:28.318590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.319788 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.319791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:28.319876 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.320378 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:28.320396 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:28.321811 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:28.324550 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.324554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.324556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.324813 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.325119 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:28.325123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:28.325125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:28.325217 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:28.325644 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:28.325648 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:28.325745 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 11:09:28.325850 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 11:09:28.325989 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 11:09:28.326171 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.314742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.16.61.60:41879)
0619 11:09:28.315118 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.315125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.315159 (+ 34us) server_negotiation.cc:407] Connection header received
0619 11:09:28.315189 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.315192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.315263 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.316312 (+ 1049us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.316451 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.317617 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.318835 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.319683 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.319786 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.321968 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.321972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.321974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.322022 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.324416 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.324921 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.324925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.324927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.325013 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.325317 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.325320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.325322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.325546 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.325636 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.325934 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.326042 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260619 11:09:28.326356 6580 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } }
I20260619 11:09:28.326510 6580 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ebb2b5961f4b18adfecff9ed34c4cc, State: Initialized, Role: FOLLOWER
I20260619 11:09:28.326766 6580 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } }
I20260619 11:09:28.327171 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.327302 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:28.327944 6580 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:28.329375 6505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68: Bootstrap starting.
I20260619 11:09:28.332016 6505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:28.332872 6505 log.cc:824] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68: Log is configured to *not* fsync() on all Append() calls
W20260619 11:09:28.333333 6686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:28.333401 6686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:28.331184 6512 master_runner.cc:186] Error getting master registration for 127.16.61.62:38499: OK, instance_id { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" instance_seqno: 1781867367850929 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:09:28.334496 6512 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:09:28.335456 6505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68: No bootstrap required, opened a new log
I20260619 11:09:28.338506 6505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } }
I20260619 11:09:28.338716 6505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7498de29f74a4c93812ec1cd40664a68, State: Initialized, Role: FOLLOWER
I20260619 11:09:28.339022 6505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } }
I20260619 11:09:28.339561 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.339865 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:28.340281 6505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:28.340358 6479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 11:09:28.340469 6479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:28.341642 6479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } }
I20260619 11:09:28.342533 6479 leader_election.cc:288] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [CANDIDATE]: Term 1 election: Requested vote from peers a8ebb2b5961f4b18adfecff9ed34c4cc (127.16.61.61:44773), 2dd6663674884be58503e60dbd7cfb71 (127.16.61.60:41879)
I20260619 11:09:28.343307 6554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7498de29f74a4c93812ec1cd40664a68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc"
I20260619 11:09:28.343542 6554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:28.344741 6554 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7498de29f74a4c93812ec1cd40664a68 in term 1.
I20260619 11:09:28.345165 6455 leader_election.cc:302] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7498de29f74a4c93812ec1cd40664a68, a8ebb2b5961f4b18adfecff9ed34c4cc; no voters:
I20260619 11:09:28.345185 6622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7498de29f74a4c93812ec1cd40664a68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2dd6663674884be58503e60dbd7cfb71"
I20260619 11:09:28.345389 6622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:28.345474 6689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:28.345757 6689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [term 1 LEADER]: Becoming Leader. State: Replica: 7498de29f74a4c93812ec1cd40664a68, State: Running, Role: LEADER
I20260619 11:09:28.346138 6689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } }
I20260619 11:09:28.346613 6622 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7498de29f74a4c93812ec1cd40664a68 in term 1.
I20260619 11:09:28.347958 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7498de29f74a4c93812ec1cd40664a68. Latest consensus state: current_term: 1 leader_uuid: "7498de29f74a4c93812ec1cd40664a68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.348124 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: This master's current role is: LEADER
W20260619 11:09:28.348440 6458 tablet.cc:2369] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:28.348603 6707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:28.351639 6707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:28.352523 6437 master_runner.cc:263] No leader master found from master 7498de29f74a4c93812ec1cd40664a68
I20260619 11:09:28.354952 6437 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:09:28.363574 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.348772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41316 (local address 127.16.61.60:41879)
0619 11:09:28.348991 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.348996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.354661 (+ 5665us) server_negotiation.cc:407] Connection header received
0619 11:09:28.354749 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.354751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.354796 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.354867 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.355808 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.356599 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.357797 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.358134 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.358204 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.360065 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.360069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.360072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.360096 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.361803 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.362284 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.362287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.362289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.362366 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.362657 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.362660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.362661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.362850 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.362977 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.363297 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.363418 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260619 11:09:28.373553 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.349389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.16.61.62:38499)
0619 11:09:28.349499 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.349503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.354889 (+ 5386us) server_negotiation.cc:407] Connection header received
0619 11:09:28.357080 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.357084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.357125 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.357171 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.359980 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.360716 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.364969 (+ 4253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.365306 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.365376 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.367003 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.367007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.367010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.367035 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.368692 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.369696 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.369700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.369702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.369782 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.370080 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.370083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.370085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.370276 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.370388 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.373166 (+ 2778us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.373323 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:09:28.376556 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:28.348046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.16.61.61:44773)
0619 11:09:28.348166 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.348171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.351962 (+ 3791us) server_negotiation.cc:407] Connection header received
0619 11:09:28.353391 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.353395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.353458 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.353509 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.358060 (+ 4551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.358822 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.364013 (+ 5191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:28.364379 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:28.364465 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:28.370684 (+ 6219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:28.370687 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:28.370690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:28.370717 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:28.372711 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.374531 (+ 1820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.374534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.374536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.374669 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:28.375093 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:28.375097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:28.375098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:28.375377 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:28.375545 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:28.376014 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 11:09:28.376377 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 11:09:28.381258 6554 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [term 1 FOLLOWER]: Refusing update from remote peer 7498de29f74a4c93812ec1cd40664a68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:28.381286 6622 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [term 1 FOLLOWER]: Refusing update from remote peer 7498de29f74a4c93812ec1cd40664a68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:28.382246 6689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:28.382647 6689 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:28.386926 6661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7498de29f74a4c93812ec1cd40664a68. Latest consensus state: current_term: 1 leader_uuid: "7498de29f74a4c93812ec1cd40664a68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.387040 6661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:28.389925 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7498de29f74a4c93812ec1cd40664a68. Latest consensus state: current_term: 1 leader_uuid: "7498de29f74a4c93812ec1cd40664a68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.390030 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:28.392237 6705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7498de29f74a4c93812ec1cd40664a68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.392351 6705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:28.393419 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7498de29f74a4c93812ec1cd40664a68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.393529 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:28.394054 6712 mvcc.cc:205] Tried to move back new op lower bound from 7298528740881768448 to 7298528740749512704. Current Snapshot: MvccSnapshot[applied={T|T < 7298528740881768448}]
I20260619 11:09:28.394202 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7498de29f74a4c93812ec1cd40664a68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.394311 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68 [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:28.394477 6715 mvcc.cc:205] Tried to move back new op lower bound from 7298528740881768448 to 7298528740749512704. Current Snapshot: MvccSnapshot[applied={T|T < 7298528740881768448}]
I20260619 11:09:28.397145 6661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7498de29f74a4c93812ec1cd40664a68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7498de29f74a4c93812ec1cd40664a68" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 38499 } } peers { permanent_uuid: "a8ebb2b5961f4b18adfecff9ed34c4cc" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 44773 } } peers { permanent_uuid: "2dd6663674884be58503e60dbd7cfb71" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 41879 } } }
I20260619 11:09:28.397250 6661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:28.397650 6713 mvcc.cc:205] Tried to move back new op lower bound from 7298528740881768448 to 7298528740749512704. Current Snapshot: MvccSnapshot[applied={T|T < 7298528740881768448}]
I20260619 11:09:28.398787 6707 catalog_manager.cc:1347] Generated new cluster ID: d6879071a77f4b74b0cb68b8a92b008b
I20260619 11:09:28.398844 6707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:28.425994 6707 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:28.427188 6707 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:28.457562 6707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7498de29f74a4c93812ec1cd40664a68: Generated new TSK 0
I20260619 11:09:28.458366 6707 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:09:28.483729 6658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.481234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41328 (local address 127.16.61.60:41879)
0619 11:09:28.481367 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.481372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.482667 (+ 1295us) server_negotiation.cc:407] Connection header received
0619 11:09:28.482869 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.482873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.482920 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.482972 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.483550 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41328: BlockingRecv error: recv got EOF from 127.0.0.1:41328 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 11:09:28.485304 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.482037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.16.61.61:44773)
0619 11:09:28.482149 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.482153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.484414 (+ 2261us) server_negotiation.cc:407] Connection header received
0619 11:09:28.484611 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.484615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.484661 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.484712 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.485192 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54396: BlockingRecv error: recv got EOF from 127.0.0.1:54396 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 11:09:28.487345 6509 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.486214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.16.61.62:38499)
0619 11:09:28.486347 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.486352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.486503 (+ 151us) server_negotiation.cc:407] Connection header received
0619 11:09:28.486631 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.486634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.486688 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.486791 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.487223 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45110: BlockingRecv error: recv got EOF from 127.0.0.1:45110 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 11:09:28.493613 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.492082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.16.61.61:44773)
0619 11:09:28.492182 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.492188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.492642 (+ 454us) server_negotiation.cc:407] Connection header received
0619 11:09:28.492857 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.492861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.492949 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.493017 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.493505 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54402: BlockingRecv error: recv got EOF from 127.0.0.1:54402 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 11:09:28.498734 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.497236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.16.61.61:44773)
0619 11:09:28.497334 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.497338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.497865 (+ 527us) server_negotiation.cc:407] Connection header received
0619 11:09:28.498050 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.498053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.498100 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.498150 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.498651 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54406: BlockingRecv error: recv got EOF from 127.0.0.1:54406 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 11:09:28.504182 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.502669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.16.61.61:44773)
0619 11:09:28.502769 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.502772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.503284 (+ 512us) server_negotiation.cc:407] Connection header received
0619 11:09:28.503470 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.503473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.503533 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.503587 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.504097 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54414: BlockingRecv error: recv got EOF from 127.0.0.1:54414 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 11:09:28.511754 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.510178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.16.61.61:44773)
0619 11:09:28.510298 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.510302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.510826 (+ 524us) server_negotiation.cc:407] Connection header received
0619 11:09:28.511013 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.511016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.511063 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.511115 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.511640 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54424: BlockingRecv error: recv got EOF from 127.0.0.1:54424 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 11:09:28.523159 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.521743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.16.61.61:44773)
0619 11:09:28.521839 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.521843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.522334 (+ 491us) server_negotiation.cc:407] Connection header received
0619 11:09:28.522500 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.522504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.522561 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.522607 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.523064 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54428: BlockingRecv error: recv got EOF from 127.0.0.1:54428 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 11:09:28.542796 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.541466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.16.61.61:44773)
0619 11:09:28.541584 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.541588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.541858 (+ 270us) server_negotiation.cc:407] Connection header received
0619 11:09:28.541978 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.541983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.542056 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.542157 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.542681 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54440: BlockingRecv error: recv got EOF from 127.0.0.1:54440 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 11:09:28.578117 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.576882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.16.61.61:44773)
0619 11:09:28.577033 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.577038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.577251 (+ 213us) server_negotiation.cc:407] Connection header received
0619 11:09:28.577383 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.577386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.577433 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.577506 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.577970 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54448: BlockingRecv error: recv got EOF from 127.0.0.1:54448 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 11:09:28.645257 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.644096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.16.61.61:44773)
0619 11:09:28.644256 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.644262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.644435 (+ 173us) server_negotiation.cc:407] Connection header received
0619 11:09:28.644541 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.644544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.644601 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.644700 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.645109 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54458: BlockingRecv error: recv got EOF from 127.0.0.1:54458 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 11:09:28.776784 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:28.775541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.16.61.61:44773)
0619 11:09:28.775734 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:28.775739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:28.775875 (+ 136us) server_negotiation.cc:407] Connection header received
0619 11:09:28.776057 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:28.776061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:28.776115 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:28.776208 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:28.776662 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54468: BlockingRecv error: recv got EOF from 127.0.0.1:54468 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 11:09:29.036198 6585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:29.035118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.16.61.61:44773)
0619 11:09:29.035257 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:29.035261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:29.035276 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:09:29.035478 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:29.035481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:29.035544 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:29.035617 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:29.036104 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54472: BlockingRecv error: recv got EOF from 127.0.0.1:54472 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 11:09:29.324887 6672 catalog_manager.cc:1259] Loaded cluster ID: d6879071a77f4b74b0cb68b8a92b008b
I20260619 11:09:29.324970 6672 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71: loading cluster ID for follower catalog manager: success
I20260619 11:09:29.326610 6672 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71: acquiring CA information for follower catalog manager: success
I20260619 11:09:29.327656 6672 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2dd6663674884be58503e60dbd7cfb71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:09:29.334429 6686 catalog_manager.cc:1259] Loaded cluster ID: d6879071a77f4b74b0cb68b8a92b008b
I20260619 11:09:29.334489 6686 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc: loading cluster ID for follower catalog manager: success
I20260619 11:09:29.336184 6686 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc: acquiring CA information for follower catalog manager: success
I20260619 11:09:29.337301 6686 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8ebb2b5961f4b18adfecff9ed34c4cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:09:29.553922 6785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:29.550241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.16.61.61:44773)
0619 11:09:29.550488 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:29.550492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:29.550507 (+ 15us) server_negotiation.cc:407] Connection header received
0619 11:09:29.550617 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:29.550619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:29.550740 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:29.550836 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:29.551061 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.551613 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.552556 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.552781 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.552864 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:29.553121 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:29.553538 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:29.553627 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:29.553684 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 11:09:29.553728 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260619 11:09:29.560499 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:29.557167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.16.61.60:41879)
0619 11:09:29.557394 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:29.557398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:29.557429 (+ 31us) server_negotiation.cc:407] Connection header received
0619 11:09:29.557509 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:29.557511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:29.557606 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:29.557689 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:29.557939 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.558440 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.559194 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.559404 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.559477 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:29.559771 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:29.560117 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:29.560205 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:29.560276 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 11:09:29.560322 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260619 11:09:29.567288 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:29.563365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.16.61.60:41879)
0619 11:09:29.563540 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:29.563544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:29.563785 (+ 241us) server_negotiation.cc:407] Connection header received
0619 11:09:29.563926 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:29.563931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:29.563981 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:29.564053 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:29.564300 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.564945 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.566061 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.566424 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.566520 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:29.566855 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:29.566923 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:29.567001 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:29.567125 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 11:09:29.567171 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 11:09:29.568213 6785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:29.563227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.16.61.61:44773)
0619 11:09:29.563496 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:29.563501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:29.564130 (+ 629us) server_negotiation.cc:407] Connection header received
0619 11:09:29.564328 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:29.564332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:29.564382 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:29.564457 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:29.564718 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.565455 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.566682 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.567002 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.567118 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:29.567751 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:29.567826 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:29.567907 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:29.568041 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 11:09:29.568090 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260619 11:09:29.568682 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:29.563183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.16.61.62:38499)
0619 11:09:29.563701 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:29.563707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:29.563723 (+ 16us) server_negotiation.cc:407] Connection header received
0619 11:09:29.563813 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:29.563816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:29.563948 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:29.564095 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:29.564568 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.565406 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.566449 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:29.566727 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:29.566819 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:29.567821 (+ 1002us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:29.568246 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:29.568368 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:29.568444 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 11:09:29.568500 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":66,"threads_started":1}
I20260619 11:09:29.574771 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6437
I20260619 11:09:29.581588 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6512
I20260619 11:09:29.587321 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6586
2026-06-19T11:09:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3059 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 11:09:29 dist-test-slave-8vcz krb5kdc[6802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 11:09:29 dist-test-slave-8vcz krb5kdc[6802](info): set up 2 sockets
Jun 19 11:09:29 dist-test-slave-8vcz krb5kdc[6802](info): commencing operation
krb5kdc: starting...
W20260619 11:09:30.660121 16628 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/test-user.keytab.
Jun 19 11:09:30 dist-test-slave-8vcz krb5kdc[6802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T11:09:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T11:09:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.61.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
Entry for principal kudu/127.16.61.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab.
I20260619 11:09:30.776856 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35385
--webserver_interface=127.16.61.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:35385,127.16.61.61:43135,127.16.61.60:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 11:09:30.884954 6815 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:09:30.885207 6815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:30.885231 6815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:30.885327 6815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:30.889230 6815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:30.889279 6815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:30.889307 6815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:30.889353 6815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:30.889389 6815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:30.895140 6815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:35385,127.16.61.61:43135,127.16.61.60:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.62.keytab
--principal=kudu/127.16.61.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.62:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.61.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:30.896198 6815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:30.897220 6815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:30.905795 6821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:30.905845 6823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:30.905771 6820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:30.906272 6815 server_base.cc:1034] running on GCE node
I20260619 11:09:30.906813 6815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:30.907986 6815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:30.909188 6815 hybrid_clock.cc:648] HybridClock initialized: now 1781867370909155 us; error 42 us; skew 500 ppm
Jun 19 11:09:30 dist-test-slave-8vcz krb5kdc[6802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867370, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:30.911947 6815 init.cc:377] Logged in from keytab as kudu/127.16.61.62@KRBTEST.COM (short username kudu)
I20260619 11:09:30.912920 6815 webserver.cc:460] Webserver started at http://127.16.61.62:43453/ using document root <none> and password file <none>
I20260619 11:09:30.913419 6815 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:30.913478 6815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:30.913635 6815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:30.915551 6815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/data/instance:
uuid: "4e3e868d074e4bba986d8e636b09676b"
format_stamp: "Formatted at 2026-06-19 11:09:30 on dist-test-slave-8vcz"
server_key: "869bbd6d086121b6b9b9dafc6b382c68"
server_key_iv: "85e9e3d0417f2ab9a0dbfaae9bb7d7e2"
server_key_version: "encryptionkey@0"
I20260619 11:09:30.916129 6815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/wal/instance:
uuid: "4e3e868d074e4bba986d8e636b09676b"
format_stamp: "Formatted at 2026-06-19 11:09:30 on dist-test-slave-8vcz"
server_key: "869bbd6d086121b6b9b9dafc6b382c68"
server_key_iv: "85e9e3d0417f2ab9a0dbfaae9bb7d7e2"
server_key_version: "encryptionkey@0"
I20260619 11:09:30.919922 6815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 11:09:30.922241 6830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:30.923532 6815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:30.923661 6815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/wal
uuid: "4e3e868d074e4bba986d8e636b09676b"
format_stamp: "Formatted at 2026-06-19 11:09:30 on dist-test-slave-8vcz"
server_key: "869bbd6d086121b6b9b9dafc6b382c68"
server_key_iv: "85e9e3d0417f2ab9a0dbfaae9bb7d7e2"
server_key_version: "encryptionkey@0"
I20260619 11:09:30.923792 6815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:30.952240 6815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:30.955444 6815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:30.955663 6815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:30.962553 6815 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.62:35385
I20260619 11:09:30.962921 6882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.62:35385 every 8 connection(s)
I20260619 11:09:30.963550 6815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/data/info.pb
I20260619 11:09:30.966223 6883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:30.969743 6883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:30.973019 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6815
I20260619 11:09:30.973150 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-0/wal/instance
I20260619 11:09:30.973449 16628 external_mini_cluster.cc:1428] Setting key acb19747224b0b9c9393f0d641120642
Jun 19 11:09:30 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867370, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.61@KRBTEST.COM" created.
I20260619 11:09:30.984498 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:30.971905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.16.61.62:35385)
0619 11:09:30.972361 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:30.972379 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:30.972569 (+ 190us) server_negotiation.cc:407] Connection header received
0619 11:09:30.973686 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:30.973690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:30.973876 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:30.974083 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:30.974861 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:30.975843 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:30.976885 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:30.977243 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:30.977416 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:30.979885 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:30.979891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:30.979905 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:30.979924 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:30.981536 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:30.982048 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:30.982054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:30.982056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:30.982159 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:30.982467 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:30.982470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:30.982471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:30.982807 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:30.982944 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:30.983287 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 11:09:30.983516 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
I20260619 11:09:30.984498 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:30.971905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35385 (local address 127.0.0.1:44330)
0619 11:09:30.972410 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:30.972454 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:30.973357 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:30.974068 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:30.974073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:30.974494 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:30.974719 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:30.974731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:30.976001 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:30.976004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:30.976730 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:30.976739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:30.977390 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:30.977394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:30.977507 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:30.978126 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:30.978145 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:30.979721 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:30.981660 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:30.981670 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:30.981678 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:30.981956 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:30.982261 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:30.982264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:30.982265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:30.982372 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:30.982919 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:30.982924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:30.983060 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 11:09:30.983217 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 11:09:30.983564 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":106,"spinlock_wait_cycles":51584,"thread_start_us":170,"threads_started":1}
I20260619 11:09:30.986826 6883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:30.987546 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:30.987275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38060)
0619 11:09:30.987405 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:30.987486 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 11:09:30.987752 6832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
I20260619 11:09:30.988612 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:30.988398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38070)
0619 11:09:30.988506 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:30.988551 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 11:09:30.988771 6883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:43135: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab.
Entry for principal kudu/127.16.61.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab.
I20260619 11:09:30.998622 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.61:43135
--webserver_interface=127.16.61.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab
--principal=kudu/127.16.61.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:35385,127.16.61.61:43135,127.16.61.60:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:09:31.038000 6883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } attempt: 1
I20260619 11:09:31.038957 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.038623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38072)
0619 11:09:31.038809 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.038875 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 11:09:31.039819 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.039532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38074)
0619 11:09:31.039696 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.039754 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 11:09:31.039963 6883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:43135: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
I20260619 11:09:31.108151 6883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } attempt: 2
I20260619 11:09:31.109319 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.108931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38082)
0619 11:09:31.109097 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.109161 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 11:09:31.110152 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.109914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38086)
0619 11:09:31.110041 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.110096 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 11:09:31.110292 6883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.61:43135: Network error: Client connection negotiation failed: client connection to 127.16.61.61:43135: connect: Connection refused (error 111)
W20260619 11:09:31.111778 6890 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:09:31.112129 6890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:31.112195 6890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:31.112282 6890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 11:09:31.116838 6890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:31.116932 6890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:31.116983 6890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:31.117048 6890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:31.117089 6890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:31.122694 6890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:35385,127.16.61.61:43135,127.16.61.60:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.61.keytab
--principal=kudu/127.16.61.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.61:43135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.61.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:31.123859 6890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:31.124897 6890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:31.133373 6895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:31.133404 6898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:31.133415 6896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:31.133739 6890 server_base.cc:1034] running on GCE node
I20260619 11:09:31.134377 6890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:31.135447 6890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:31.136647 6890 hybrid_clock.cc:648] HybridClock initialized: now 1781867371136635 us; error 61 us; skew 500 ppm
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:31.139524 6890 init.cc:377] Logged in from keytab as kudu/127.16.61.61@KRBTEST.COM (short username kudu)
I20260619 11:09:31.140528 6890 webserver.cc:460] Webserver started at http://127.16.61.61:34291/ using document root <none> and password file <none>
I20260619 11:09:31.141149 6890 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:31.141244 6890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:31.141479 6890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:31.143483 6890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/data/instance:
uuid: "deb521996c8f4b759959671118798440"
format_stamp: "Formatted at 2026-06-19 11:09:31 on dist-test-slave-8vcz"
server_key: "5287ba4e9b7b4a0536fdd35a8059ef0b"
server_key_iv: "8ced87c31197113c76b3ad45c852f1c0"
server_key_version: "encryptionkey@0"
I20260619 11:09:31.144177 6890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/wal/instance:
uuid: "deb521996c8f4b759959671118798440"
format_stamp: "Formatted at 2026-06-19 11:09:31 on dist-test-slave-8vcz"
server_key: "5287ba4e9b7b4a0536fdd35a8059ef0b"
server_key_iv: "8ced87c31197113c76b3ad45c852f1c0"
server_key_version: "encryptionkey@0"
I20260619 11:09:31.147753 6890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 11:09:31.150066 6905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:31.151225 6890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 11:09:31.151376 6890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/wal
uuid: "deb521996c8f4b759959671118798440"
format_stamp: "Formatted at 2026-06-19 11:09:31 on dist-test-slave-8vcz"
server_key: "5287ba4e9b7b4a0536fdd35a8059ef0b"
server_key_iv: "8ced87c31197113c76b3ad45c852f1c0"
server_key_version: "encryptionkey@0"
I20260619 11:09:31.151499 6890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:31.164111 6890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:31.166872 6890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:31.167068 6890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:31.174198 6890 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.61:43135
I20260619 11:09:31.174485 6957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.61:43135 every 8 connection(s)
I20260619 11:09:31.175249 6890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/data/info.pb
I20260619 11:09:31.177929 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:31.181608 6958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:31.184844 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6890
I20260619 11:09:31.185001 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-1/wal/instance
I20260619 11:09:31.185315 16628 external_mini_cluster.cc:1428] Setting key 78ad9064b151602f1cd7f970aa73c521
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.61.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.61.60@KRBTEST.COM" created.
I20260619 11:09:31.197738 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.183505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.16.61.62:35385)
0619 11:09:31.183690 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.183696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.184542 (+ 846us) server_negotiation.cc:407] Connection header received
0619 11:09:31.185785 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.185789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.185840 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.185919 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.187505 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.188396 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.190271 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.190512 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.190629 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.193631 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.193635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.193637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.193662 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.195574 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.196112 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.196117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.196119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.196200 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.196547 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.196553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.196554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.196749 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.196879 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.197429 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.197559 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:09:31.198556 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.183919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35385 (local address 127.0.0.1:44346)
0619 11:09:31.184354 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.184419 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:31.185543 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:31.186126 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:31.186138 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:31.186683 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:31.187321 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.187337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.188556 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.188562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.190115 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.190132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.190711 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.190717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.190964 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.191690 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:31.191720 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:31.193439 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:31.195711 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.195720 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.195736 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.196001 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.196297 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.196300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.196302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.196444 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.196891 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:31.196899 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:31.197169 (+ 270us) client_negotiation.cc:769] Sending connection context
0619 11:09:31.197356 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 11:09:31.197700 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260619 11:09:31.200016 6958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:31.201484 6883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } attempt: 3
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867370, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
I20260619 11:09:31.212692 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.201886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38102)
0619 11:09:31.201985 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.202000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:31.202070 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:31.202519 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:31.202524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:31.202701 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:31.202814 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.202821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.203650 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.203656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.204205 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.204213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.204786 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.204789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.204903 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.205375 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:31.205394 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:31.207820 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:31.210956 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.210959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.210961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.211160 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.211584 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.211587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.211588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.211725 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.212230 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:31.212233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:31.212316 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 11:09:31.212451 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 11:09:31.212567 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 11:09:31.212989 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.201957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.16.61.61:43135)
0619 11:09:31.202288 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.202297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.202309 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:09:31.202339 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.202342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.202378 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.202467 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.202942 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.203547 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.204322 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.204670 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.204784 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.207972 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.207980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.207991 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.208012 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.210854 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.211329 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.211336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.211338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.211445 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.211809 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.211811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.211813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.212122 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.212258 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.212542 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.212736 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"spinlock_wait_cycles":3968,"thread_start_us":125,"threads_started":1}
I20260619 11:09:31.213200 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.200588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38088)
0619 11:09:31.200992 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.201007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:31.201094 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:31.201483 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:31.201486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:31.201674 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:31.201783 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.201794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.203219 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.203224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.204160 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.204174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.204945 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.204949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.205061 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.205538 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:31.205548 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:31.208355 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:31.211265 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.211267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.211269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.211450 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.211874 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.211880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.211882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.211964 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.212372 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:31.212375 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:31.212490 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 11:09:31.212640 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 11:09:31.213062 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":302,"spinlock_wait_cycles":1408}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.61.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab.
Entry for principal kudu/127.16.61.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab.
I20260619 11:09:31.212989 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.200678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.16.61.61:43135)
0619 11:09:31.200992 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.201004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.201067 (+ 63us) server_negotiation.cc:407] Connection header received
0619 11:09:31.201201 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.201211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.201355 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.201492 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.201901 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.203089 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.204302 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.204714 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.204842 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.208511 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.208515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.208517 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.208545 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.210873 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.211542 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.211547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.211549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.211629 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.212057 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.212060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.212062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.212270 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.212379 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.212766 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.212855 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":217,"spinlock_wait_cycles":24832,"thread_start_us":142,"threads_started":1}
I20260619 11:09:31.214404 16628 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIllhcy/build/debug/bin/kudu
/tmp/dist-test-taskIllhcy/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.60:43235
--webserver_interface=127.16.61.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab
--principal=kudu/127.16.61.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.61.20:38821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.61.62:35385,127.16.61.61:43135,127.16.61.60:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 11:09:31.215138 6883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:31.215410 6958 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:31.216101 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.215707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52532)
0619 11:09:31.215954 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.216010 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260619 11:09:31.216178 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.215896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52546)
0619 11:09:31.216037 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.216106 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 11:09:31.216364 6907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
I20260619 11:09:31.216847 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.216567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52552)
0619 11:09:31.216732 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.216782 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260619 11:09:31.216987 6883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:43235: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
I20260619 11:09:31.217518 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.217319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52556)
0619 11:09:31.217419 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.217462 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 11:09:31.217669 6958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:43235: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
I20260619 11:09:31.248148 6883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } attempt: 1
I20260619 11:09:31.249023 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.248734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52564)
0619 11:09:31.248882 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.248947 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 11:09:31.249743 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.249510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52570)
0619 11:09:31.249635 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.249687 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 11:09:31.249869 6883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:43235: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
I20260619 11:09:31.266921 6958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } attempt: 1
I20260619 11:09:31.267974 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.267589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52584)
0619 11:09:31.267782 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.267861 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 11:09:31.268939 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.268626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52600)
0619 11:09:31.268764 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.268840 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 11:09:31.269176 6958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:43235: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
I20260619 11:09:31.325042 6883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } attempt: 2
I20260619 11:09:31.325929 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.325615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52616)
0619 11:09:31.325771 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.325851 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 11:09:31.326242 6965 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 11:09:31.326562 6965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 11:09:31.326610 6965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 11:09:31.326666 6965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 11:09:31.326738 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.326452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52628)
0619 11:09:31.326562 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.326639 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 11:09:31.326861 6883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:43235: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
W20260619 11:09:31.330987 6965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 11:09:31.331074 6965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 11:09:31.331120 6965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 11:09:31.331178 6965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 11:09:31.331215 6965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 11:09:31.336549 6965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.61.20:38821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.61.62:35385,127.16.61.61:43135,127.16.61.60:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/krb5kdc/kudu_127.16.61.60.keytab
--principal=kudu/127.16.61.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.61.60:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.61.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 10:20:32 UTC on ccycloud.quasar-txpsjl.root.comops.site
build id 4455439
I20260619 11:09:31.337404 6958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } attempt: 2
I20260619 11:09:31.337736 6965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 11:09:31.338207 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.337934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52642)
0619 11:09:31.338075 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.338134 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 11:09:31.338961 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.338724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52646)
0619 11:09:31.338849 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.338897 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 11:09:31.339005 6965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 11:09:31.339123 6958 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.61.60:43235: Network error: Client connection negotiation failed: client connection to 127.16.61.60:43235: connect: Connection refused (error 111)
W20260619 11:09:31.347201 6971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:31.347247 6974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 11:09:31.347201 6972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 11:09:31.347497 6965 server_base.cc:1034] running on GCE node
I20260619 11:09:31.348083 6965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 11:09:31.349120 6965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 11:09:31.350291 6965 hybrid_clock.cc:648] HybridClock initialized: now 1781867371350258 us; error 29 us; skew 500 ppm
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 11:09:31.353129 6965 init.cc:377] Logged in from keytab as kudu/127.16.61.60@KRBTEST.COM (short username kudu)
I20260619 11:09:31.354041 6965 webserver.cc:460] Webserver started at http://127.16.61.60:42797/ using document root <none> and password file <none>
I20260619 11:09:31.354676 6965 fs_manager.cc:359] Metadata directory not provided
I20260619 11:09:31.354775 6965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 11:09:31.355007 6965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 11:09:31.357131 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/data/instance:
uuid: "99a19bf022df4d6399a5e3ad83df7636"
format_stamp: "Formatted at 2026-06-19 11:09:31 on dist-test-slave-8vcz"
server_key: "df62e88eab28046e2f5542270b9093b8"
server_key_iv: "0fe5833eff6280a6c2f35b63e00d8f42"
server_key_version: "encryptionkey@0"
I20260619 11:09:31.357704 6965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/wal/instance:
uuid: "99a19bf022df4d6399a5e3ad83df7636"
format_stamp: "Formatted at 2026-06-19 11:09:31 on dist-test-slave-8vcz"
server_key: "df62e88eab28046e2f5542270b9093b8"
server_key_iv: "0fe5833eff6280a6c2f35b63e00d8f42"
server_key_version: "encryptionkey@0"
I20260619 11:09:31.361375 6965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 11:09:31.363799 6981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 11:09:31.364967 6965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 11:09:31.365108 6965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/data,/tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/wal
uuid: "99a19bf022df4d6399a5e3ad83df7636"
format_stamp: "Formatted at 2026-06-19 11:09:31 on dist-test-slave-8vcz"
server_key: "df62e88eab28046e2f5542270b9093b8"
server_key_iv: "0fe5833eff6280a6c2f35b63e00d8f42"
server_key_version: "encryptionkey@0"
I20260619 11:09:31.365217 6965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 11:09:31.386343 6965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 11:09:31.389631 6965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 11:09:31.389827 6965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 11:09:31.397153 6965 rpc_server.cc:307] RPC server started. Bound to: 127.16.61.60:43235
I20260619 11:09:31.397437 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.61.60:43235 every 8 connection(s)
I20260619 11:09:31.398296 6965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/data/info.pb
I20260619 11:09:31.401037 7034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 11:09:31.401284 16628 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIllhcy/build/debug/bin/kudu as pid 6965
I20260619 11:09:31.401468 16628 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIllhcy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781867094678158-16628-0/minicluster-data/master-2/wal/instance
I20260619 11:09:31.401770 16628 external_mini_cluster.cc:1428] Setting key f548c2a481022e44057f680d21bab992
I20260619 11:09:31.405990 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:31.416529 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.403062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.16.61.60:43235)
0619 11:09:31.403454 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.403468 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.403503 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:09:31.404247 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.404264 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.404604 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.404875 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.405850 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.406928 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.407989 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.408579 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.408743 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.411404 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.411409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.411417 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.411442 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.413290 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.413735 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.413746 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.413750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.413899 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.414203 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.414205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.414206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.414740 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.414981 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.415207 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.415487 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":129,"threads_started":1}
I20260619 11:09:31.417316 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.407806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44354 (local address 127.16.61.62:35385)
0619 11:09:31.407958 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.407963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.408341 (+ 378us) server_negotiation.cc:407] Connection header received
0619 11:09:31.408471 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.408474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.408530 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.408622 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.409485 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.409953 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.411024 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.411402 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.411487 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.413850 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.413853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.413856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.413882 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.415487 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.415970 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.415974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.415976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.416063 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.416346 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.416348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.416350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.416598 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.416693 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.417103 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.417203 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 11:09:31.417315 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.407752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.62:35385 (local address 127.0.0.1:44354)
0619 11:09:31.408203 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.408251 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:31.408371 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:31.408699 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:31.408705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:31.409163 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:31.409352 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.409361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.410078 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.410085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.410893 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.410910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.411512 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.411518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.411714 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.412169 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:31.412188 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:31.413715 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:31.415594 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.415601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.415605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.415877 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.416161 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.416163 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.416165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.416248 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.416751 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:31.416756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:31.416868 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 11:09:31.417001 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 11:09:31.417158 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260619 11:09:31.418479 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.62@KRBTEST.COM
I20260619 11:09:31.426023 6883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } attempt: 3
I20260619 11:09:31.427944 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.419196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.16.61.61:43135)
0619 11:09:31.419322 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.419326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.419346 (+ 20us) server_negotiation.cc:407] Connection header received
0619 11:09:31.419382 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.419384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.419419 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.419481 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.419961 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.420756 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.421817 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.422098 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.422182 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.424218 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.424221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.424224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.424247 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.425827 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.426344 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.426348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.426350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.426436 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.426795 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.426798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.426802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.427052 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.427108 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.427527 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.427695 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 11:09:31.428407 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.418813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.16.61.62:35385)
0619 11:09:31.418918 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.418922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.418959 (+ 37us) server_negotiation.cc:407] Connection header received
0619 11:09:31.419092 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.419098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.419149 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.419221 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.420000 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.420756 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.422271 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.422461 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.422572 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.425087 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.425097 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.425100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.425120 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.426700 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.427185 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.427188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.427190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.427271 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.427532 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.427537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.427539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.427789 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.427993 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.428132 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.428237 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 11:09:31.428442 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.418946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.61:43135 (local address 127.0.0.1:38108)
0619 11:09:31.419089 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.419102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:31.419167 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:31.419530 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:31.419535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:31.419735 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:31.419849 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.419856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.420872 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.420876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.421690 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.421700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.422197 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.422199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.422267 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.422728 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:31.422739 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:31.424085 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:31.425918 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.425920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.425921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.426230 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.426554 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.426557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.426559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.426652 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.427200 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:31.427202 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:31.427319 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 11:09:31.428177 (+ 858us) client_negotiation.cc:240] Negotiation successful
0619 11:09:31.428299 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 11:09:31.429953 7034 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } has no permanent_uuid. Determining permanent_uuid...
I20260619 11:09:31.430295 6958 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } attempt: 3
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867370, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.62@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.61.61@KRBTEST.COM
I20260619 11:09:31.435963 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.426611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.16.61.60:43235)
0619 11:09:31.426723 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.426728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.426739 (+ 11us) server_negotiation.cc:407] Connection header received
0619 11:09:31.426770 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.426774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.426812 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.426885 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.427307 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.428932 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.429904 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.430211 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.430832 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.432630 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.432636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.432640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.432664 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.434191 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.434703 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.434710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.434713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.434787 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.435125 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.435128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.435131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.435316 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.435452 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.435738 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.435837 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:09:31.435963 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.426370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52660)
0619 11:09:31.426506 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.426522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:31.426607 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:31.426921 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:31.426924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:31.427082 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:31.427204 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.427209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.429029 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.429033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.429813 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.429821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.430535 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.430542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.430636 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.431053 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:31.431064 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:31.432491 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:31.434304 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.434307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.434310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.434573 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.434904 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.434907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.434909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.434998 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.435450 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:31.435453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:31.435555 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 11:09:31.435668 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 11:09:31.435808 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.60@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
Jun 19 11:09:31 dist-test-slave-8vcz krb5kdc[6802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781867371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.61.61@KRBTEST.COM for kudu/127.16.61.60@KRBTEST.COM
I20260619 11:09:31.439424 6883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b: Bootstrap starting.
I20260619 11:09:31.440174 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.430399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52672)
0619 11:09:31.430505 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.430517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:31.430586 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:31.431579 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:31.431581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:31.431764 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:31.431858 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.431867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.432957 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.432960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.433820 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.433831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.434485 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.434491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.434568 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.434963 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:31.434976 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:31.436843 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:31.438723 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.438726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.438728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.438966 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.439226 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.439230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.439232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.439313 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.439661 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:31.439687 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:31.439767 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 11:09:31.439880 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 11:09:31.440006 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 11:09:31.440219 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.430416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.16.61.60:43235)
0619 11:09:31.431336 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.431345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.431358 (+ 13us) server_negotiation.cc:407] Connection header received
0619 11:09:31.431389 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.431391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.431436 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.431967 (+ 531us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.432037 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.432871 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.433949 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.434373 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.434754 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.436962 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.436965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.436967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.436988 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.438606 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.439061 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.439064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.439068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.439136 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.439398 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.439404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.439406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.439569 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.439670 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.439923 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.440058 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":844,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260619 11:09:31.440806 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.429449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.16.61.61:43135)
0619 11:09:31.429537 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.429541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.429553 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:09:31.429651 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.429653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.429689 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.429755 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.430587 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.431287 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.432364 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.432677 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.432742 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.437307 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.437314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.437318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.437348 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.438899 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.439358 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.439362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.439363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.439447 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.439805 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.439808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.439809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.440004 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.440157 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.440546 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.440662 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 11:09:31.442888 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636: Bootstrap starting.
I20260619 11:09:31.443362 6883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:31.443848 16628 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 11:09:31.444268 6883 log.cc:824] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:31.445302 7034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:31.444916 6857 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4e3e868d074e4bba986d8e636b09676b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44362
I20260619 11:09:31.446053 7034 log.cc:824] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:31.446434 6883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b: No bootstrap required, opened a new log
I20260619 11:09:31.447949 7034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636: No bootstrap required, opened a new log
I20260619 11:09:31.449345 6883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } }
I20260619 11:09:31.449589 6883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3e868d074e4bba986d8e636b09676b, State: Initialized, Role: FOLLOWER
I20260619 11:09:31.449906 6883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } }
I20260619 11:09:31.450989 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.430653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.61.60:43235 (local address 127.0.0.1:52674)
0619 11:09:31.430776 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 11:09:31.430792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 11:09:31.430857 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 11:09:31.431653 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 11:09:31.431656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 11:09:31.431844 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 11:09:31.431925 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.431931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.433371 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.433374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.434313 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 11:09:31.434328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.435032 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.435036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 11:09:31.435120 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.435522 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 11:09:31.435535 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 11:09:31.441261 (+ 5726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 11:09:31.449317 (+ 8056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.449320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.449322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.449519 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.449901 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 11:09:31.449904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 11:09:31.449906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 11:09:31.449994 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 11:09:31.450478 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 11:09:31.450482 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 11:09:31.450592 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 11:09:31.450687 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 11:09:31.450821 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 11:09:31.451282 6883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:31.451386 6857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 11:09:31.451534 6857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:31.451637 7034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } }
I20260619 11:09:31.451887 7034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a19bf022df4d6399a5e3ad83df7636, State: Initialized, Role: FOLLOWER
I20260619 11:09:31.452054 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.431072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.16.61.60:43235)
0619 11:09:31.431423 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.431432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.431444 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:09:31.431485 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.431488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.431527 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.432026 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.432149 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.433237 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.434479 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.434892 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.435336 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.444698 (+ 9362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.444702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.444705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.444727 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.449207 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.449677 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.449683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.449685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.449817 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.450179 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.450182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.450184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.450382 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.450474 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.450798 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.450974 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":280,"thread_start_us":80,"threads_started":1}
I20260619 11:09:31.452195 7034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } }
I20260619 11:09:31.452875 6857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } }
I20260619 11:09:31.453411 7034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:31.453727 6857 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [CANDIDATE]: Term 1 election: Requested vote from peers deb521996c8f4b759959671118798440 (127.16.61.61:43135), 99a19bf022df4d6399a5e3ad83df7636 (127.16.61.60:43235)
I20260619 11:09:31.455116 6958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440: Bootstrap starting.
W20260619 11:09:31.457247 6836 tablet.cc:2369] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 11:09:31.456017 7008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e3e868d074e4bba986d8e636b09676b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99a19bf022df4d6399a5e3ad83df7636"
I20260619 11:09:31.457461 7008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [term 0 FOLLOWER]: Advancing to term 1
W20260619 11:09:31.458366 7066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:31.458425 7066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:31.458434 7008 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e3e868d074e4bba986d8e636b09676b in term 1.
I20260619 11:09:31.458652 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.458756 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:31.458817 6832 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e3e868d074e4bba986d8e636b09676b, 99a19bf022df4d6399a5e3ad83df7636; no voters:
I20260619 11:09:31.458976 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.459086 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:31.459160 7043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 11:09:31.459232 7043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [term 1 LEADER]: Becoming Leader. State: Replica: 4e3e868d074e4bba986d8e636b09676b, State: Running, Role: LEADER
I20260619 11:09:31.459506 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } }
I20260619 11:09:31.459987 6958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440: Neither blocks nor log segments found. Creating new log.
I20260619 11:09:31.460695 6958 log.cc:824] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440: Log is configured to *not* fsync() on all Append() calls
I20260619 11:09:31.461172 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3e868d074e4bba986d8e636b09676b. Latest consensus state: current_term: 1 leader_uuid: "4e3e868d074e4bba986d8e636b09676b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.461309 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:31.457067 6815 master_runner.cc:186] Error getting master registration for 127.16.61.61:43135: OK, instance_id { permanent_uuid: "deb521996c8f4b759959671118798440" instance_seqno: 1781867371172332 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:09:31.462316 6815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:09:31.462675 6958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440: No bootstrap required, opened a new log
W20260619 11:09:31.465227 7055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:31.465299 7055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:31.466710 6958 raft_consensus.cc:348] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } }
I20260619 11:09:31.466921 6958 raft_consensus.cc:729] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb521996c8f4b759959671118798440, State: Initialized, Role: FOLLOWER
I20260619 11:09:31.467224 6958 consensus_queue.cc:260] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } }
I20260619 11:09:31.464543 6965 master_runner.cc:186] Error getting master registration for 127.16.61.61:43135: OK, instance_id { permanent_uuid: "deb521996c8f4b759959671118798440" instance_seqno: 1781867371172332 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 11:09:31.467803 6965 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 11:09:31.472563 6958 sys_catalog.cc:564] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 11:09:31.472591 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e3e868d074e4bba986d8e636b09676b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "deb521996c8f4b759959671118798440"
I20260619 11:09:31.472818 6932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [term 0 FOLLOWER]: Advancing to term 1
I20260619 11:09:31.474130 6932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e3e868d074e4bba986d8e636b09676b in term 1.
I20260619 11:09:31.474902 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.475037 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 11:09:31.478140 7088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 11:09:31.478211 7088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 11:09:31.480875 7075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 11:09:31.481582 7075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 11:09:31.486399 7008 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [term 1 FOLLOWER]: Refusing update from remote peer 4e3e868d074e4bba986d8e636b09676b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:31.487838 6932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [term 1 FOLLOWER]: Refusing update from remote peer 4e3e868d074e4bba986d8e636b09676b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 11:09:31.488407 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:31.491907 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3e868d074e4bba986d8e636b09676b. Latest consensus state: current_term: 1 leader_uuid: "4e3e868d074e4bba986d8e636b09676b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.492013 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:31.492187 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 11:09:31.495273 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3e868d074e4bba986d8e636b09676b. Latest consensus state: current_term: 1 leader_uuid: "4e3e868d074e4bba986d8e636b09676b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.495373 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:31.497251 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.476659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.16.61.62:35385)
0619 11:09:31.476764 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.476769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.478573 (+ 1804us) server_negotiation.cc:407] Connection header received
0619 11:09:31.478770 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.478773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.478820 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.478870 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.479870 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.480761 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.482925 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.483260 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.483333 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.493142 (+ 9809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.493148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.493152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.493179 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.494942 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.495439 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.495442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.495445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.495529 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.495885 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.495891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.495892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.496125 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.496256 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.496947 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.497070 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 11:09:31.500821 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.476340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.16.61.60:43235)
0619 11:09:31.476462 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.476466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.484265 (+ 7799us) server_negotiation.cc:407] Connection header received
0619 11:09:31.485518 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.485521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.485568 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.485622 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.488436 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.489218 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.491700 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.492110 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.492200 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.497243 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.497246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.497248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.497274 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.498803 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.499491 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.499495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.499496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.499570 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.499904 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.499907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.499909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.500107 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.500213 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.500567 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.500677 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 11:09:31.504943 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e3e868d074e4bba986d8e636b09676b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.506019 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e3e868d074e4bba986d8e636b09676b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.506129 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:31.506639 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:31.506794 7093 mvcc.cc:205] Tried to move back new op lower bound from 7298528753601421312 to 7298528753501286400. Current Snapshot: MvccSnapshot[applied={T|T < 7298528753601421312}]
I20260619 11:09:31.506961 7097 mvcc.cc:205] Tried to move back new op lower bound from 7298528753601421312 to 7298528753501286400. Current Snapshot: MvccSnapshot[applied={T|T < 7298528753601421312}]
I20260619 11:09:31.508421 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:31.475952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.16.61.61:43135)
0619 11:09:31.476071 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.476075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.483913 (+ 7838us) server_negotiation.cc:407] Connection header received
0619 11:09:31.484365 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.484368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.484416 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.484465 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.486549 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.487285 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.490362 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:31.490749 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:31.490827 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:31.495291 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 11:09:31.495296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 11:09:31.495298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 11:09:31.495323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 11:09:31.497102 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.499459 (+ 2357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.499466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.499470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.499560 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 11:09:31.499914 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 11:09:31.499917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 11:09:31.499919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 11:09:31.500107 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 11:09:31.500212 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:31.508158 (+ 7946us) server_negotiation.cc:299] Negotiation successful
0619 11:09:31.508277 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 11:09:31.510325 7075 catalog_manager.cc:1347] Generated new cluster ID: 8d5427162fc34e4bb468141f4587b853
I20260619 11:09:31.510378 7075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 11:09:31.515154 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e3e868d074e4bba986d8e636b09676b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.515254 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b [sys.catalog]: This master's current role is: LEADER
I20260619 11:09:31.516331 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e3e868d074e4bba986d8e636b09676b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3e868d074e4bba986d8e636b09676b" member_type: VOTER last_known_addr { host: "127.16.61.62" port: 35385 } } peers { permanent_uuid: "deb521996c8f4b759959671118798440" member_type: VOTER last_known_addr { host: "127.16.61.61" port: 43135 } } peers { permanent_uuid: "99a19bf022df4d6399a5e3ad83df7636" member_type: VOTER last_known_addr { host: "127.16.61.60" port: 43235 } } }
I20260619 11:09:31.516431 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 11:09:31.517119 7095 mvcc.cc:205] Tried to move back new op lower bound from 7298528753601421312 to 7298528753501286400. Current Snapshot: MvccSnapshot[applied={T|T < 7298528753601421312}]
I20260619 11:09:31.531237 7075 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 11:09:31.532418 7075 catalog_manager.cc:1504] Loading token signing keys...
I20260619 11:09:31.547741 7075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e3e868d074e4bba986d8e636b09676b: Generated new TSK 0
I20260619 11:09:31.548496 7075 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 11:09:31.586627 6885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.583808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.16.61.62:35385)
0619 11:09:31.583932 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.583937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.585604 (+ 1667us) server_negotiation.cc:407] Connection header received
0619 11:09:31.585794 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.585797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.585846 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.585898 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.586494 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51552: BlockingRecv error: recv got EOF from 127.0.0.1:51552 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 11:09:31.588385 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.584526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.16.61.61:43135)
0619 11:09:31.584632 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.584636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.587247 (+ 2611us) server_negotiation.cc:407] Connection header received
0619 11:09:31.587693 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.587696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.587745 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.587798 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.588272 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34084: BlockingRecv error: recv got EOF from 127.0.0.1:34084 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 11:09:31.596323 7039 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.585107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.16.61.60:43235)
0619 11:09:31.585234 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.585239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.587441 (+ 2202us) server_negotiation.cc:407] Connection header received
0619 11:09:31.587555 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.587558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.587618 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.587734 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.596211 (+ 8477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57192: BlockingRecv error: recv got EOF from 127.0.0.1:57192 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 11:09:31.600625 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.599206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.16.61.61:43135)
0619 11:09:31.599308 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.599312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.599743 (+ 431us) server_negotiation.cc:407] Connection header received
0619 11:09:31.599956 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.599959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.600034 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.600086 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.600535 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34100: BlockingRecv error: recv got EOF from 127.0.0.1:34100 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 11:09:31.605612 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.604140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.16.61.61:43135)
0619 11:09:31.604237 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.604242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.604768 (+ 526us) server_negotiation.cc:407] Connection header received
0619 11:09:31.604945 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.604948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.604995 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.605047 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.605526 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34114: BlockingRecv error: recv got EOF from 127.0.0.1:34114 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 11:09:31.611168 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.609768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.16.61.61:43135)
0619 11:09:31.609855 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.609860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.610315 (+ 455us) server_negotiation.cc:407] Connection header received
0619 11:09:31.610530 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.610533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.610576 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.610629 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.611081 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34128: BlockingRecv error: recv got EOF from 127.0.0.1:34128 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 11:09:31.618978 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.617423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.16.61.61:43135)
0619 11:09:31.617564 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.617568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.618093 (+ 525us) server_negotiation.cc:407] Connection header received
0619 11:09:31.618281 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.618286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.618334 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.618381 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.618881 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34134: BlockingRecv error: recv got EOF from 127.0.0.1:34134 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 11:09:31.631115 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.629431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.16.61.61:43135)
0619 11:09:31.629556 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.629561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.630117 (+ 556us) server_negotiation.cc:407] Connection header received
0619 11:09:31.630325 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.630329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.630383 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.630440 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.631007 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34144: BlockingRecv error: recv got EOF from 127.0.0.1:34144 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 11:09:31.651631 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.649823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.16.61.61:43135)
0619 11:09:31.649936 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.649943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.650533 (+ 590us) server_negotiation.cc:407] Connection header received
0619 11:09:31.650752 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.650755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.650809 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.650861 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.651507 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34152: BlockingRecv error: recv got EOF from 127.0.0.1:34152 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 11:09:31.687825 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.686454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.16.61.61:43135)
0619 11:09:31.686575 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.686580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.686862 (+ 282us) server_negotiation.cc:407] Connection header received
0619 11:09:31.686986 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.686989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.687062 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.687164 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.687703 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34160: BlockingRecv error: recv got EOF from 127.0.0.1:34160 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 11:09:31.755231 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.753966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.16.61.61:43135)
0619 11:09:31.754123 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.754126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.754283 (+ 157us) server_negotiation.cc:407] Connection header received
0619 11:09:31.754373 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.754375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.754425 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.754567 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.755050 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34174: BlockingRecv error: recv got EOF from 127.0.0.1:34174 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 11:09:31.886440 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:31.885370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34176 (local address 127.16.61.61:43135)
0619 11:09:31.885524 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:31.885527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:31.885685 (+ 158us) server_negotiation.cc:407] Connection header received
0619 11:09:31.885807 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:31.885810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:31.885859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:31.885951 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:31.886336 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34176: BlockingRecv error: recv got EOF from 127.0.0.1:34176 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 11:09:32.145632 6963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 11:09:32.144706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.16.61.61:43135)
0619 11:09:32.144844 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:32.144848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:32.144953 (+ 105us) server_negotiation.cc:407] Connection header received
0619 11:09:32.145025 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:32.145028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:32.145078 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:32.145162 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 11:09:32.145537 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34182: BlockingRecv error: recv got EOF from 127.0.0.1:34182 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 11:09:32.459690 7066 catalog_manager.cc:1259] Loaded cluster ID: 8d5427162fc34e4bb468141f4587b853
I20260619 11:09:32.459774 7066 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636: loading cluster ID for follower catalog manager: success
I20260619 11:09:32.461439 7066 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636: acquiring CA information for follower catalog manager: success
I20260619 11:09:32.462569 7066 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99a19bf022df4d6399a5e3ad83df7636: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:09:32.479290 7088 catalog_manager.cc:1259] Loaded cluster ID: 8d5427162fc34e4bb468141f4587b853
I20260619 11:09:32.479386 7088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440: loading cluster ID for follower catalog manager: success
I20260619 11:09:32.480983 7088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440: acquiring CA information for follower catalog manager: success
I20260619 11:09:32.482079 7088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P deb521996c8f4b759959671118798440: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 11:09:32.663218 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:32.659741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.16.61.61:43135)
0619 11:09:32.659959 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:32.659963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:32.659998 (+ 35us) server_negotiation.cc:407] Connection header received
0619 11:09:32.660142 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:32.660150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:32.660280 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:32.660415 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:32.660620 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.661132 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.661910 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.662137 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.662221 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:32.662473 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:32.662898 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:32.662986 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:32.663053 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 11:09:32.663098 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260619 11:09:32.669538 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:32.666105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.16.61.60:43235)
0619 11:09:32.666326 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:32.666329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:32.666362 (+ 33us) server_negotiation.cc:407] Connection header received
0619 11:09:32.666432 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:32.666434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:32.666537 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:32.666624 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:32.666795 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.667295 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.668099 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.668376 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.668488 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:32.668778 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:32.669143 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:32.669267 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:32.669330 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 11:09:32.669380 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260619 11:09:32.676537 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:32.672350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.16.61.61:43135)
0619 11:09:32.672559 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:32.672564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:32.672909 (+ 345us) server_negotiation.cc:407] Connection header received
0619 11:09:32.673110 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:32.673113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:32.673160 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:32.673208 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:32.673587 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.674325 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.675339 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.675686 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.675776 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:32.676070 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:32.676141 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:32.676210 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:32.676350 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 11:09:32.676396 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 11:09:32.676589 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:32.672961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.16.61.60:43235)
0619 11:09:32.673064 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:32.673070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:32.673084 (+ 14us) server_negotiation.cc:407] Connection header received
0619 11:09:32.673118 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:32.673121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:32.673160 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:32.673232 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:32.673500 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.674096 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.675377 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.675649 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.675767 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:32.676142 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:32.676214 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:32.676279 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:32.676417 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 11:09:32.676463 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 11:09:32.677178 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 11:09:32.672348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.16.61.62:35385)
0619 11:09:32.672618 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 11:09:32.672621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 11:09:32.672633 (+ 12us) server_negotiation.cc:407] Connection header received
0619 11:09:32.672770 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 11:09:32.672774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 11:09:32.672930 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 11:09:32.673039 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 11:09:32.673431 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.674309 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.675475 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 11:09:32.675848 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 11:09:32.675951 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 11:09:32.676293 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 11:09:32.676588 (+ 295us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 11:09:32.676675 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 11:09:32.676953 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 11:09:32.677014 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
I20260619 11:09:32.679823 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6815
I20260619 11:09:32.686198 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6890
I20260619 11:09:32.691975 16628 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIllhcy/build/debug/bin/kudu with pid 6965
2026-06-19T11:09:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3101 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6161 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (278009 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST